Intelligent repurposable cooling systems for mobile datacenter

ABSTRACT

A cooling system for a datacenter is disclosed. An evaporative cooling subsystem provides blown air for cooling the datacenter and a repurposable refrigerant cooling subsystem controls moisture of the blown air in a first configuration and independently cools the datacenter in a second configuration.

FIELD

At least one embodiment pertains to cooling systems for a datacenter. Inat least one embodiment, an evaporative cooling subsystem provides blownair for cooling the datacenter and a repurposable refrigerant coolingsubsystem controls moisture of the blown air in a first configurationand independently cools the datacenter in a second configuration.

BACKGROUND

Datacenter cooling systems typically use fans to circulate air throughserver components. Certain supercomputers or other high capacitycomputers may use water or other cooling systems than air coolingsystems to draw heat away from the server components or racks of thedatacenter to an area external to the datacenter. The cooling systemsmay include a chiller within the datacenter area. The area external tothe datacenter may be a cooling tower or other external heat exchangerthat receives heated coolant from the datacenter and disperses the heatby forced air or other means to the environment (or an external coolingmedium) before the cooled coolant is recirculated back into thedatacenter. In an example, the chiller and the cooling tower togetherform a chilling facility with pumps responsive to temperature measuredby external devices applied to the datacenter. Air cooling systems alonemay not draw sufficient heat to support effective or efficient coolingin datacenters and liquid cooling systems may not be economical for thedemands of the datacenter.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 is a block diagram of an example datacenter having a coolingsystem subject to improvements described in at least one embodiment;

FIG. 2A is a block diagram illustrating mobile datacenter features of acooling system incorporating a repurposable refrigerant coolingsubsystem, according to at least one embodiment;

FIG. 2B is a block diagram illustrating an evaporative cooling subsystembenefiting from a repurposable refrigerant cooling subsystem, accordingto at least one embodiment;

FIG. 2C is a block diagram illustrating datacenter-level features of acooling system incorporating an evaporative cooling subsystem along witha repurposable refrigerant cooling subsystem, according to at least oneembodiment;

FIG. 3A is a block diagram illustrating rack-level and server-levelfeatures of a cooling system incorporating an evaporative coolingsubsystem that is supported by a repurposable refrigerant coolingsubsystem, according to at least one embodiment;

FIG. 3B is a block diagram illustrating rack-level and server-levelfeatures of a cooling system incorporating a liquid-based coolingsubsystem of the multiple mode cooling subsystem, according to at leastone embodiment;

FIG. 4A is a block diagram illustrating rack-level and server-levelfeatures of a cooling system incorporating a repurposable refrigerantcooling subsystem of the multiple mode cooling subsystem, according toat least one embodiment;

FIG. 4B is a block diagram illustrating rack-level and server-levelfeatures of a cooling system incorporating a dielectric-based coolingsubsystem of the multiple mode cooling subsystem, according to at leastone embodiment;

FIG. 5 is a process flow of steps available for a method of using ormaking the cooling system of FIGS. 2A-4B and 6A-17D, according to atleast one embodiment;

FIG. 6A illustrates an example datacenter, in which at least oneembodiment from FIGS. 2A-5 may be used;

FIGS. 6B, 6C illustrate inference and/or training logic, such as used inFIG. 6A and in at least one embodiment of the present disclosure, forenabling and/or supporting a multiple mode cooling subsystem having arepurposable refrigerant cooling subsystem, according to variousembodiments;

FIG. 7A is a block diagram illustrating an exemplary computer system,which may be a system with interconnected devices and components, asystem-on-a-chip (SOC) or some combination thereof formed with aprocessor that may include execution units to execute an instruction tosupport and/or to enable a multiple mode cooling subsystem having arepurposable refrigerant cooling subsystem as described herein,according to at least one embodiment;

FIG. 7B is a block diagram illustrating an electronic device forutilizing a processor to support and/or to enable a multiple modecooling subsystem having a repurposable refrigerant cooling subsystemas, according to at least one embodiment;

FIG. 7C is a block diagram illustrating an electronic device forutilizing a processor to support and/or to enable a multiple modecooling subsystem having a repurposable refrigerant cooling subsystemas, according to at least one embodiment;

FIG. 8 illustrates a further example computer system, according to atleast one embodiment, to implement various processes and methods for amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem as described throughout this disclosure;

FIG. 9A illustrates an exemplary architecture in which GPUs arecommunicatively coupled to multi-core processors over high-speed linksfor enabling and/or supporting a multiple mode cooling subsystem havinga repurposable refrigerant cooling subsystem, according to at least oneembodiment of the disclosure herein;

FIG. 9B illustrates additional details for an interconnection between amulti-core processor and a graphics acceleration module in accordancewith one exemplary embodiment;

FIG. 9C illustrates another exemplary embodiment in which acceleratorintegration circuit is integrated within a processor for enabling and/orsupporting a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem, according to at least one embodiment ofthe disclosure herein;

FIG. 9D illustrates an exemplary accelerator integration slice 990 forenabling and/or supporting a multiple mode cooling subsystem having arepurposable refrigerant cooling subsystem, according to at least oneembodiment of the disclosure herein;

FIG. 9E illustrates additional details for one exemplary embodiment of ashared model. to enable and/or support a multiple mode cooling subsystemhaving a repurposable refrigerant cooling subsystem, according to atleast one embodiment of the disclosure herein;

FIG. 9F illustrates additional details for one exemplary embodiment of aunified memory, addressable via a common virtual memory address spaceused to access physical processor memories and GPU memories to enableand/or support a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem, according to at least one embodiment ofthe disclosure herein;

FIG. 10A illustrates exemplary integrated circuits and associatedgraphics processors, according to embodiments described herein for amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem as;

FIGS. 10B-10C illustrate exemplary integrated circuits and associatedgraphics processors, according to at least one embodiment, to supportand/or to enable a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem;

FIGS. 10D-10E illustrate additional exemplary graphics processor logic,according to at least one embodiment, to support and/or to enable amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem as;

FIG. 11A is a block diagram illustrating a computing system to supportand/or to enable a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem, according to at least one embodiment;

FIG. 11B illustrates a parallel processor to support and/or to enable amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem according to at least one embodiment;

FIG. 11C is a block diagram of a partition unit according to at leastone embodiment;

FIG. 11D shows a graphics multiprocessor used for a multiple modecooling subsystem having a repurposable refrigerant cooling subsystemaccording to at least one embodiment;

FIG. 11E shows a graphics multiprocessor according to at least oneembodiment;

FIG. 12A illustrates a multi-GPU computing system, according to at leastone embodiment;

FIG. 12B is a block diagram of a graphics processor, according to atleast one embodiment;

FIG. 13 is a block diagram illustrating micro-architecture for aprocessor that may include logic circuits to perform instructions,according to at least one embodiment;

FIG. 14 illustrates a deep learning application processor, according toat least one embodiment;

FIG. 15 is a block diagram of a neuromorphic processor, according to atleast one embodiment;

FIG. 16A is a block diagram of a processing system, according to atleast one embodiment;

FIG. 16B is a block diagram of a processor having one or more processorcores, an integrated memory controller, and an integrated graphicsprocessor, according to at least one embodiment;

FIG. 16C is a block diagram of hardware logic of a graphics processorcore, according to at least one embodiment;

FIGS. 16D-16E illustrate thread execution logic including an array ofprocessing elements of a graphics processor core according to at leastone embodiment;

FIG. 17A illustrates a parallel processing unit, according to at leastone embodiment;

FIG. 17B illustrates a general processing cluster, according to at leastone embodiment;

FIG. 17C illustrates a memory partition unit of a parallel processingunit, in accordance with at least one embodiment; and

FIG. 17D illustrates a streaming multi-processor, according to at leastone embodiment.

DETAILED DESCRIPTION

Air cooling of high density servers is inefficient and ineffective inview of the high heat requirements caused by present day computingcomponents. However, the requirements are subject to change or tend torange from a minimum to a maximum of different cooling requirements. Thedifferent cooling requirements also reflect different heat features ofthe datacenter. In at least one embodiment, heat generated from thecomponents, servers, and racks are cumulatively referred to as a heatfeature or a cooling requirement as the cooling requirement must addressthe heat feature entirely. In at least one embodiment, the heat featureor the cooling requirement for a cooling system is the heat generated orthe cooling requirement of the components, servers, or racks associatedwith the cooling system and may be a portion of components, servers, andracks in the datacenter. For purposes of economies, providing aircooling along or providing liquid cooling alone may not address presentday datacenter cooling requirements. Further, providing liquid coolingwhen heat generated from a server is at a minimum of the differentcooling requirements taxes components of a liquid cooling system andalso wastes energy requirements to enable the liquid cooling system.Still further, the ability to address the different cooling requirementsis challenging within a mobile datacenter because of lack of space tosupport cooling systems able to provide a variety of different coolingmedia.

As such, the present disclosure seeks prospects in a cooling system fora datacenter having a repurposable refrigerant cooling subsystem in aform factor for one or more racks of the datacenter. The one or moreracks fit within a container or trailer that also includes the serverracks of a mobile datacenter. The repurposable refrigerant coolingsubsystem has a refrigerant cooling subsystem that is repurposable tocontrol moisture in an evaporative cooling subsystem in a firstconfiguration and to provide independent cooling of the datacenter in asecond configuration. As such, the repurposable refrigerant coolingsubsystem has two or more different cooling systems, and is adjustableto different cooling requirements of the datacenter within differentcooling capacities offered by at least the evaporative cooling subsystemfunctioning together with the repurposable refrigerant cooling subsystemin the first configuration and the repurposable refrigerant coolingsubsystem functioning independently in a second configuration to offerat least two different cooling subsystems.

In at least one embodiment, additional cooling subsystems may be offeredwith the at least two different cooling subsystems. In at least oneembodiment, the cooling system of the present disclosure is for coolingcomputing components such as a graphics processing unit (GPU), a centralprocessing unit (CPU), or switching components. These computingcomponents are used in servers assembled in server trays on racks in adatacenter. As the computing components are miniaturized by technologyadvances, the server trays and the racks accommodate more and morecomputing components, thereby requiring dissipation of more heatgenerated per component than in prior systems. However, it is also thecase that computation and involvement of the computing components toperform functions cause the dissipation of more heat than when thecomputing components are idle. One issue addressed in the presentdisclosure is an inability to address basic cooling requirements byair-base cooling systems because of the ineffectiveness of such a systemand because of the unwieldy nature of evaporative cooling.

In at least one embodiment, a datacenter liquid cooling system issupported by a chiller plant or system that may expensive because it isdesigned with over-provisioning requirements. The over-provisioningrequirements may be due to lack of proper control for removal of heatfrom the server, such as from one or more components (such as GPU, CPU,etc.), from the collective components in the server boxes, or thecollective components within the racks. Efficient use ofhigh-thermal-capacity liquid cooling systems may be over shadowed by themany intermediate features in the cooling system, include liquidchillers, pumps, coolant distribution units (CDUs), and heat exchangers.The repurposable refrigerant cooling subsystem of a datacenter coolingsystem of the present disclosure is able to provide different coolingcapacities that at least perform dual-functions, such as to controlhumidity and to address the different cooling requirements using hightemperature facility water, in at least one embodiment that may be usedas coolant in the primary and the secondary cooling loops forliquid-based cooling of high heat components and systems. A refrigerantcooling subsystem is repurposable for controlling humidity of anair-based subsystem relying on evaporative cooling as well as forindependent cooling, while functioning as a repurposable refrigerantcooling subsystem. The cooling system includes a dielectric-basedcooling subsystem for immersion cooling; and the evaporative coolingsubsystem, depending on the different cooling requirements. In additionto performing dual purposes of controlling humidity of the evaporativecooling subsystem and providing independent cooling, the repurposablerefrigerant cooling subsystem supplements the evaporative coolingsubsystem that relies on evaporative cooling, but also enables removalof regular to extreme heat by an additional use of a dielectricrefrigerant that support the dielectric-based cooling subsystem.

In at least one embodiment, an evaporative cooling subsystem providesblown air for cooling the datacenter. The blow air is provided inaccordance to a first heat feature in the datacenter. In at least oneembodiment, the first heat feature is a lowest heat feature is addressedby an evaporative cooling subsystem formed of the evaporative coolingsubsystem. In addition to addressing the lowest heat feature of thedatacenter, the evaporative cooling subsystem also reflects the mosteconomical cooling subsystem of the datacenter cooling system. In atleast one embodiment, the evaporative cooling subsystem works oractivates concurrently with a repurposable refrigerant (or otherrepurposable refrigerant) cooling subsystem to control moisture of theblown air in a first configuration of the repurposable refrigerantcooling subsystem. In at least one embodiment, the repurposablerefrigerant cooling subsystem may be activated independently to cool thedatacenter in a second configuration and according to a second heatfeature of the datacenter. In the second configuration, the evaporativecooling subsystem may not be in use, in at least one embodiment.

In at least one embodiment, the repurposable refrigerant coolingsubsystem includes an evaporator coil associated with a condensing unitprovided external to the datacenter. A refrigerant changes phase fromliquid to vapor in an evaporator to reach cooler temperatures withincirculating pipes (or after absorbing heat from the environment), whichenables the repurposable refrigerant cooling. Once the vapor is expendedin dissipating cooler temperatures (or absorbing the heat), it iscompressed and condensed in a condensing unit to a liquid phase and sentback to the evaporator. In at least one embodiment, the condensing unitis as close to a secondary cooling loop as possible. In at least oneembodiment, the condensing unit is also as close to the evaporativecooling system as possible to ensure timely and efficient removal ofmoisture in the evaporative cooling subsystem. In at least oneembodiment, the secondary cooling loop for each of a liquid-based andthe repurposable refrigerant cooling subsystems are neighborly, withintheir own pipes and manifolds. In at least one embodiment, thecondensing unit is located on a roof area of the datacenter or acontainer (of a mobile datacenter) and is able to directly access roomor row manifolds to provide refrigerant in a refrigerant cooling loop ofa repurposable refrigerant cooling subsystem and for controllingmoisture in the evaporative cooling subsystem. In at least onembodiment, the evaporator coil of the repurposable refrigerant coolingsubsystem enables a direct expansion refrigerant for the refrigerationcooling loop and is co-existent with a secondary cooling loop carryingcoolant, also referred to as the warm temperature liquid cooling system.

In at least one embodiment, the evaporative cooling subsystem includesits own condenser coil and its own evaporator coil to condition blownair for the datacenter using refrigerant from the repurposablerefrigerant cooling subsystem. The refrigerant from the repurposablerefrigerant cooling subsystem enters a condenser coil of the evaporativecooling subsystem under high temperature and pressure, and is cooled byscavenger air that flows through the evaporative cooling subsystem. Thehigh pressure refrigerant is cooler now and in liquid phase as it passesthrough an evaporator coil just before the blown air enters thedatacenter. The moisture in the blow air is condensed at the evaporatorcoil. The blown air, conditioned by at least removal of moisture to adetermined threshold, flows into the datacenter to address a first heatfeature or cooling requirement in the datacenter. In at least oneembodiment, the spent refrigerant may be compressed by a compressorwithin the evaporative cooling subsystem before being returned to therepurposable refrigerant cooling subsystem. In at least one embodiment,the refrigerant of the evaporative subsystem is circulated in theevaporative subsystem and augmented as required from the repurposablerefrigerant cooling subsystem.

A server box or server tray may include the above-referenced computingcomponents, but multiple server boxes, at a rack level or multiple ofracks benefit from the repurposable refrigerant cooling subsystem of thepresent disclosure. The repurposable refrigerant cooling subsystem,along with the evaporative cooling subsystem, is able to address thedifferent cooling requirements from the cooling subsystems fitted withina rack or multiple racks as a form factor of the cooling system. Thiseliminates additional costs to operate a datacenter chiller with moreefficiency than in cooler or nominal environmental temperatures, wherean evaporative cooling system is sufficient. In at least one embodiment,miniature pumps or other flow control devices form the one or more flowcontrollers that are sensitive to the thermal requirements of at leastone or more sensors in the server or the racks for the repurposablerefrigerant cooling subsystem.

FIG. 1 is a block diagram of an example datacenter 100 having a coolingsystem subject to improvements described in at least one embodiment. Thedatacenter 100 may be one or more rooms 102 having racks 110 andauxiliary equipment to house one or more servers on one or more servertrays. The datacenter 100 is supported by a cooling tower 104 locatedexternal to the datacenter 100. The cooling tower 104 dissipates heatfrom within the datacenter 100 by acting on a primary cooling loop 106.Further, a cooling distribution unit (CDU) 112 is used between theprimary cooling loop 106 and a second or secondary cooling loop 108 toenable extraction of the heat from the second or secondary cooling loop108 to the primary cooling loop 106. The secondary cooling loop 108 isable to access various plumbing all the way into the server tray asrequired, in an aspect. The loops 106, 108 are illustrated as linedrawings, but a person of ordinary skill would recognize that one ormore plumbing features may be used. In an instance, flexible polyvinylchloride (PVC) pipes may be used along with associated plumbing to movethe fluid along in each of the loops 106, 108. One or more coolantpumps, in at least one embodiment, may be used to maintain pressuredifferences within the loops 106, 108 to enable the movement of thecoolant according to temperature sensors in various locations, includingin the room, in one or more racks 110, and/or in server boxes or servertrays within the racks 110.

In at least one embodiment, the coolant in the primary cooling loop 106and in the secondary cooling loop 108 may be at least water and anadditive, for instance, glycol or propylene glycol. In operation, eachof the primary and the secondary cooling loops has their own coolant. Inan aspect, the coolant in the secondary cooling loops may be proprietaryto requirements of the components in the server tray or racks 110. TheCDU 112 is capable of sophisticated control of the coolants,independently or concurrently, in the loops 106, 108. For instance, theCDU may be adapted to control the flow rate so that the coolant(s) isappropriately distributed to extract heat generated within the racks110. Further, more flexible tubing 114 is provided from the secondarycooling loop 108 to enter each server tray and to provide coolant to theelectrical and/or computing components. In the present disclosure, theelectrical and/or computing components are used interchangeably to referto the heat-generating components that benefit from the presentdatacenter cooling system. The tubing 118 that form part of thesecondary cooling loop 108 may be referred to as room manifolds.Separately, the tubing 116 extending from tubing 118 may also be part ofthe secondary cooling loop 108 but may be referred to as row manifolds.The tubing 114 enters the racks as part of the secondary cooling loop108, but may be referred to as rack cooling manifold. Further, the rowmanifolds 116 extend to all racks along a row in the datacenter 100. Theplumbing of the secondary cooling loop 108, including the manifolds 118,116, and 114 may be improved by at least one embodiment of the presentdisclosure. An optional chiller 120 may be provided in the primarycooling loop within datacenter 102 to support cooling before the coolingtower. To the extent additional loops exist in the primary control loop,a person of ordinary skill would recognize reading the presentdisclosure that the additional loops provide cooling external to therack and external to the secondary cooling loop; and may be takentogether with the primary cooling loop for this disclosure.

In at least one embodiment, in operation, heat generated within servertrays of the racks 110 may be transferred to a coolant exiting the racks110 via flexible tubing of the row manifold 114 of the second coolingloop 108. Pertinently, second coolant (in the secondary cooling loop108) from the CDU 112, for cooling the racks 110, moves towards theracks 110. The second coolant from the CDU 112 passes from on one sideof the room manifold having tubing 118, to one side of the rack 110 viarow manifold 116, and through one side of the server tray via tubing114. Spent second coolant (or exiting second coolant carrying the heatfrom the computing components) exits out of another side of the servertray (such as enter left side of the rack and exits right side of therack for the server tray after looping through the server tray orthrough components on the server tray). The spent second coolant thatexits the server tray or the rack 110 comes out of different side (suchas exiting side) of tubing 114 and moves to a parallel, but also exitingside of the row manifold 116. From the row manifold 116, the spentsecond coolant moves in a parallel portion of the room manifold 118going in the opposite direction than the incoming second coolant (whichmay also be the renewed second coolant), and towards the CDU 112.

In at least one embodiment, the spent second coolant exchanges its heatwith a primary coolant in the primary cooling loop 106 via the CDU 112.The spent second coolant is renewed (such as relatively cooled whencompared to the temperature at the spent second coolant stage) and readyto be cycled back to through the second cooling loop 108 to thecomputing components. Various flow and temperature control features inthe CDU 112 enable control of the heat exchanged from the spent secondcoolant or the flow of the second coolant in and out of the CDU 112. CDU112 is also able to control a flow of the primary coolant in primarycooling loop 106. As such, it is possible that some components withinthe servers and the racks do not receive the level of coolant requiredas the second or secondary loop generally provides coolant at itsdefault temperature properties based in part on the temperature sensorsthat may be in the servers and the racks. Further, because of arestriction in space in at least mobile datacenters, the requirementsherein address multiple purposes of both cooling and feature control byone or more of the cooling subsystem, such as by the repurposablerefrigerant cooling subsystem.

FIG. 2A is a block diagram illustrating mobile datacenter features of acooling system 200 incorporating a repurposable refrigerant coolingsubsystem within a rack or a multi-rack format, multiple mode coolingsubsystem 212A; 212B; 212C, according to at least one embodiment. FIG.2A may be a topological illustration of containers or pods 204A-D of amobile datacenter system or cooling system 200 that are configured in anarrangement that enables circulation of cooling media associated with amobile cooling system, according to at least one embodiment. In at leastone embodiment, the mobile datacenter cooling system 200 is configuredwith multiple containers or pods 204A-D having racks 216 (one group ofracks is referenced). In at least one embodiment, the racks 216 arecooled via one or more cooling media provided from a respective one ofcontainer manifolds 206 (one container manifold having one or moremanifolds for one or more different media is referenced). In at leastone embodiment, blown air may be delivered directly through ports forthe containers 204A, B, C, or D; or for racks 216. In addition, thecooling media passes from the container manifolds 206 to respective oneof row manifolds 208 (one set of row manifolds having inlet and outletmanifolds therein for one or more cooling media is referenced).

In at least one embodiment, each of the container manifolds 216 extendsthrough the perimeter of its respective container 204A-D. As such, therow manifolds 208 may not be required in at least one embodiment. In atleast one embodiment, the container manifolds 206 hosts respective CDUsor is coupled to a respective CDU of a multiple mode cooling subsystem212A, B from a respective one or more containers 204A-D. In at least oneembodiment, a CDU may reside in and distribute cooling media associatedwith one or more containers 204A-D, or each of the one or morecontainers 204A-D may have its own CDU (illustrated by referencenumerals 212C, which forms multiple mode cooling subsystem having aCDU). In at least one embodiment, when the mobile datacenter features ofa cooling system 200 includes a multi-mode (or multiple mode) coolingsubsystem that uses coolant, refrigerant, and blown air, the blockhaving reference numeral 212A; B may be referred to as the multi-mode(or multiple mode) cooling subsystem and may include a CDU within it tohandle features associated with a liquid-based cooling subsystem, suchas a coolant-based cooling subsystem. As such, reference to a CDU withina multiple mode cooling subsystem 212A; B; C does not limit thefunctionality of the block illustrated by reference numeral 212A; B to aCDU functionality, but at least refers to one function within the block212A; B. The blocks 212A; B; C may be referred to as a CDU, in at leastone embodiment, but may be a multiple mode cooling subsystem unlessindicated otherwise.

In at least one embodiment, the cooling system 200 is on one or moretrailer-beds. In at least one embodiment, the trailer-bed or each of thecontainers (such as containers 204A, 204D) have space for aninfrastructure area 202A, B. In at least one embodiment, the coolingsystem 200 is supported by a cooling tower 218 that is located in theinfrastructure area 202A, on its own trailer-bed, or shared betweeninfrastructure areas 202A, B of multiple containers or trailer-beds. Assuch, in at least one embodiment, each container 204A-D, as well as theinfrastructure areas 202A, B are on independent trailers or a singletrailer. In at least one embodiment, individual trailer-beds may belocated adjacent each other at a deployment location. In at least oneembodiment, transportation of the cooling system 200 is dimensionedaccording to standard allowances (such as trucking allowances by theDepartment of Transportation).

In at least one embodiment, each container 204A-D may be adapted toreceive a primary cooling loop via a respective external piping 214A; B,from a respective one or more CDUs of the multiple cooling subsystem212A, B. The respective one or more CDUs of the multiple mode coolingsubsystems 212A, B are coupled to a secondary cooling loop to reach theracks 216. In at least one embodiment, the CDUs work together as part ofa singular multiple mode cooling subsystem 212A, B, where control of thedifferent cooling media is offered by a distributed or centralizedcontrol system. The secondary cooling loop is associated with the one ormore external cooling towers 218B via the CDUs 212A, B, which sitbetween the primary and secondary cooling loops to allow a first coolingmedia of the secondary cooling loop to exchange heat with a secondcooling media of the primary cooling loop. In at least one embodiment,the containers 204A-D may be adapted to receive a primary cooling loopindirectly via the respective external piping 214A; B. The respectiveexternal piping 214A; B is part of the secondary cooling loop.

While the cooling tower 218B of FIG. 2A is illustrated as a singulartower, each CDU 212A; 212 B may be associated with its own coolingtower. The primary cooling loop may have different configurations, suchas, from a CDU 212A to each set of racks in containers 204A, 204B; orfrom a CDU 212A to racks in one container 204A, with a separate CDUcoupled to racks in the neighboring container 204B; or from CDU 212A toracks in all of the containers 204A-D, with the additional CDU 212Bconfigured for redundancy in the event of failure of a primary CDU 212A.The primary cooling loop may include additional sub-loops, such as loopshaving the row manifold 208, but the primary cooling loop may extendfully to the row manifolds by providing a singular coolant through thecontainer manifolds 206 and the row manifolds 208. In at least oneembodiment, the CDU is located close to (such in enclosed containerarea, which also referred to as infrastructure area 202A; 202B) and orwithin one or more of the containers 204A-D. The cooling tower 218B maybe located external to the container or enclosed areas. In at least oneembodiment, also as illustrated in the example of FIG. 2, the coolingtower 218B may be located on top of one or more containers 204A-D. In atleast one embodiment, one or more CDUs may be designated entirely in onecontainer to support racks of multiple containers. The CDUs may becoupled to at least a sized cooling tower located on top of thecontainer. The combination of the cooling tower on the top of thecontainer and the CDUs within the container form a mobile cooling systemthat may be co-located with adjacent trailers having only containerswith racks.

In at least one embodiment, a condenser unit 218A, having a compressorand a condenser, is located external to one or more of the containers204A-D. In at least one embodiment, the condenser unit 218A enables therepurposable refrigerant cooling subsystem within a rack or a multi-rackformat multiple cooling subsystem 212C of the present disclosure. Thecondenser unit 218A directly circulates, via lines 210B independent fromthe coolant lines 210A of the secondary cooling loop, refrigerantrequired for the repurposable refrigerant cooling subsystem within arack or a multi-rack format multiple cooling subsystem 212C of thepresent disclosure.

In at least one embodiment, the condenser unit 218A also circulaterefrigerant directly to the evaporative cooling subsystem within therack or the multi-rack format multiple cooling subsystem 212C of thepresent disclosure. In at least one embodiment, each of the rack or themulti-rack format multiple cooling subsystem 212C is independent andaddresses heat features or cooling requirements of the server racksassociated within its container. In at least one embodiment, thecondenser unit 218A is located on top of the container. In at least oneembodiment, the location of the condenser unit 218A is to ensure that itis as close to the repurposable refrigerant cooling subsystem that maybe in the form factor of one or more racks and located within thecontainer 204A; B; C; or D. In at least one embodiment, the repurposablerefrigerant cooling subsystem is located in the infrastructure area202A; 202B, adjacent to the CDU 212A; 212B.

In at least one embodiment, the condenser unit 218A may be located ontop of the infrastructure area 202A and can circulate refrigerantdirectly to the evaporative cooling subsystem and directly to therepurposable refrigerant cooling subsystem within the rack or themulti-rack format multiple cooling subsystem 212A. As such, the blockforming reference numeral 212A is referred to as a multi-mode (ormultiple mode) cooling subsystem in at least one embodiment. Asillustrated in at least FIG. 2C, in at least one embodiment, themulti-mode (or multiple mode) cooling subsystem 212A is a rack or amultiple rack format structure within the container or in aninfrastructure area. In at least one embodiment, when a multi-mode (ormultiple mode) cooling subsystem is used for cooling a datacenter thatis mobile or fixed, it may include a CDU within it to handle featuresassociated with a liquid-based cooling subsystem, such as acoolant-based cooling subsystem.

In at least one embodiment, each of the rack or the multi-rack formatmultiple cooling subsystem 212C is independent and addresses heatfeatures or cooling requirements of the server racks associated withinits container. In at least one embodiment, the condenser unit 218A islocated on top of the container. In at least one embodiment, thelocation of the condenser unit 218A is to ensure that it is as close tothe repurposable refrigerant cooling subsystem that may be in the formfactor of one or more racks and located within the container 204A; B; C;or D. In at least one embodiment, the repurposable refrigerant coolingsubsystem is located in the infrastructure area 202A; 202B, adjacent tothe CDU in the multi-mode (or multiple mode) cooling subsystem 212A;212B. In at least one embodiment, the refrigerant may reach theevaporative cooling subsystem and the repurposable refrigerant coolingsubsystem directly via lines 210B or may reach these subsystems vialines in the container manifold 206.

In at least one embodiment, FIG. 2A also illustrates that multipletrailer-beds (each hosting a container and the infrastructure area orjust the container) located adjacent to each other to enable couplingand sharing of cooling resources, such as the CDU 212A; 212B, thecondenser unit 218A, and the cooling tower 218B. In at least oneembodiment, a single trailer-bed having capacity to handle multiplecontainers or pods, as well as the cooling towers, may be illustrated inthe configuration of FIG. 2A. In either case, the containers or pods204A-D share at least cooling resources from at least two CDUs 212A; B.In at least one embodiment, each primary cooling loop from the coolingtower 218B terminates at a CDU 212A; B within its own container or otherenclosure 204A or in an adjacent container 204A; D. A secondary coolingloop extends through the container manifolds 206 to provide coolantdirectly to the racks, or indirectly through row manifolds 208. Thecoolant associated with the containers in the secondary cooling loopcirculates through both the CDUs of the respective containers 204A; D,while containers 204B; C may not have CDUs. Alternatively, in at leastone embodiment, the secondary cooling loop may include additional CDUsin containers 204B; C, with the coolant associated with the CDU ofcontainers 204A; D cooling a coolant associated with the CDUs incontainers 204B; C. The container manifolds 206 may be coupled to eachother or adjacent ones of the container manifolds via fluid couplers.

In at least one embodiment, the primary cooling loop terminates in arespective CDU of a respective container 204A-D. In at least oneembodiment, the primary cooling loop extends from the cooling tower 218Bto the CDU 212A in or adjacent to container 204A, where a dividingcoupler allows the coolant of the primary cooling loop to extend to aCDU in container 204B. In at least one embodiment, further piping of thecontainer manifolds 206 enable the primary cooling loop of one coolingtower 218B to extend through the CDUs of each container 204A-D, such asCDU 212A. Then the row manifolds 208 form the secondary cooling loop inthis case. Separately or concurrently, in at least this or in oneembodiment, CDU 212B may be used to address the cooling requirements ofthe containers 204A-D via the container manifolds 206 forming asecondary cooling loop with the row manifolds 208, while the coolingtower 218B is in a primary cooling loop with the CDU 212B. This enablesredundancy plans in the event that one of the cooling towers fails in atleast one embodiment having multiple cooling towers.

In at least one embodiment, the configuration of FIG. 2A is enabled byat least one processor that may include at least one logic unit havingtrained neural networks or other machine learning models that is trainedto determine a cooling subsystem to meet the temperature requirements,such as a heat feature or a cooling requirement, of a mobile datacenterto be deployed in response to a customer requirement. The neural networkor other machine learning models may be trained initially and may betrained further, on an on-going basis, to improve its accuracy. The atleast one processor may be provided to train one or more neural networkhaving hidden layers of neurons for evaluating temperatures withinservers or one or more racks in the datacenter with flow rates ofdifferent cooling media from at least the evaporative cooling subsystemand the repurposable refrigerant cooling subsystem.

In at least one embodiment, the at least one processor may provideoutputs associated with at least two temperatures to facilitate movementof two cooling media (such as the blown air and refrigerant,concurrently or independently) from the cooling system by controllingflow controllers associated with the two cooling media to address thefirst heat feature and the second heat feature. The at least oneprocessor is further described elsewhere in this disclosure withreference to at least the processors and neural network training schemesin FIGS. 14 and 15. The training may also to be used to evaluate flowrates or flow volumes of the two cooling media based in part on theirassociations with the temperature requirements.

In at least one embodiment, the condenser unit 218A addressesrefrigerant returned from one or more repurposable refrigerant coolingsubsystems in the multiple mode cooling subsystem 212A; B; and C. In atleast one embodiment, separate condenser units 218A may be provided toaddress refrigerants from more than one repurposable refrigerant coolingsubsystems. In at least one embodiment, the one or more repurposablerefrigerant cooling subsystems are controlled from a centralized ordistributed control system. In at least one embodiment, a condenser unitmay be placed atop the infrastructure area to address refrigerantrequirements for both the evaporative cooling subsystem and the one ormore repurposable refrigerant-based cooling subsystems, all of which areeither working together or independently via controls for associatedflow controllers of the evaporative cooling subsystem and the one ormore repurposable refrigerant-based cooling subsystems.

In at least one embodiment, the at least one processor may be trainedusing prior or test values for the heat features or coolingrequirements, and the flow rates or flow volumes of one or more coolingmedia. In at least one embodiment, the at least one processor, trainedto correlate the heat features or the cooling requirements with the flowrates or flow volumes of the one or more cooling media is able to assertoutputs having instructions to one or more flow controllers. In at leastone embodiment, one or more first flow controllers of the cooling systemis instructed to maintain a first cooling media associated with theevaporative cooling subsystem at a first flow rate and to maintain asecond cooling media associated with the repurposable refrigerantcooling subsystem at a second flow rate. The first cooling media isflown air and the second cooling media is refrigerant, in at least oneembodiment. The first flow rate for the evaporative cooling subsystem isbased in part on the first heat feature. The second flow rate associatedwith the repurposable refrigerant cooling subsystem is a flow raterequired for refrigerant to flow from the repurposable refrigerantcooling subsystem to components of the evaporative cooling subsystem tocontrol the humidity in blow air. In at least one embodiment, this setup of at least the second flow rate of the refrigerant to therepurposable refrigerant cooling subsystem is associated with a firstconfiguration of the repurposable refrigerant cooling subsystem.Furthermore, in the first configuration, the one or more first flowcontrollers are at least associated with a circulation of flow of therefrigerant to the evaporative cooling subsystem for purposes ofcontrolling humidity of the blown air.

In at least one embodiment, one or more second flow controllers are usedto maintain the second cooling media, associated with the repurposablerefrigerant cooling subsystem, at a third flow rate. The third flow rateis based in part on the second heat feature. Further, the one or morefirst flow controllers are deactivated based in part on the second heatfeature. In at least one embodiment, the third flow rate is at leastpart of a second configuration for the repurposable refrigerant coolingsubsystem. The refrigerant of the repurposable refrigerant coolingsubsystem flows to the rack, server, or datacenter area requiringcooling in the second configuration for the repurposable refrigerantcooling subsystem. As such, the one or more second flow controllers maybe different than the one or more first flow controllers so as tocirculate the refrigerant from the repurposable refrigerant coolingsubsystem to the datacenter for cooling of components, servers, andracks in the datacenter. In at least one embodiment, multi-directionalflow controllers may be used instead of the first and the second flowcontrollers to adjust the refrigerant flow in the first configuration,at the second flow rate for humidity control; and to adjust therefrigerant flow in the second configuration, at the third flow rate forcooling in the datacenter.

When a trained neural network is then applied to address at least a heatfeature or cooling requirement (also referred to as a temperaturerequirement) associated with the racks 216 in containers 204A-D, outputsmay be provided of the required flow rates or flow volumes ofrefrigerant and of other cooling media (such as blown air) required toaddress the temperature requirement. In at least one embodiment, theoutput is an extrapolation of the prior or test values provided tocontrol flow rates of various cooling media to address the heat featureor the cooling requirement, and where the neural network determines anerror in attaining cooling is the least in the configuration determinedby the output. The error is addressed by repeated back-propagation, inat least one embodiment, so that the neural network is more accurate indetermining a response to the heat feature or the cooling requirement.

In at least one embodiment, the datacenter cooling system 200 includesthe at least one processor in its racks 204A-D for continuouslyimproving the neural network based in part on current informationavailable from the deployed mobile datacenter cooling system 200. The atleast one processor executes a machine learning model to performfunctions. One function is to process the temperatures within servers orone or more racks 216 in the datacenter 200 using multiple neuron levelsof the machine learning model. The multiple neuron levels of the machinelearning model have the temperatures and have prior associated flowrates for the two cooling media for the temperatures. The temperaturesand the associated flow rates may be from prior actual coolingapplications using the configuration of the cooling system 200 or fromtests performed using the configuration of the cooling system 200. In atleast one embodiment, another function of the machine learning model isto provide the outputs associated with flow rates for the two coolingmedia to the flow controllers. The outputs are provided after anevaluation of the prior associated flow rates of individual coolingmedia of the two cooling media and prior associated temperatures.

In at least one embodiment, the evaluation attempts to correlate(achieve a zero error during training to reflect a fully classifiedmachine learning model) one or more of the flow rates of the individualcooling media to achieve a drop in temperature in the datacenter from aninitial temperature causing a flow of the individual cooling media tooccur to a second temperature causing the flow to seize. In at least oneembodiment, the cooling media is blown air and a refrigerant, where therefrigerant is required to address additional humidity that may exist inthe blown air. The datacenter 200 may be adapted to operate with atmaximum an 60% humidity, and the flow rate of the refrigerant is atleast maintained along with the flow rate of the blown air to ensurethat sufficient moisture is removed from the blown air to retain thehumidity in the datacenter at the adapted maximum limit.

In at least one embodiment, the at least one processor of the coolingsystem 200 includes instruction outputs to communicate the outputsassociated with the flow controllers to enable the first flow rate ofthe first individual cooling media used to address the first heatfeature in the datacenter, while maintaining a second flow rate of asecond individual cooling media to control humidity of the firstindividual cooling media. The instruction outputs also communicate theoutputs having instructions to deactivate the first individual coolingmedia and to enable a third flow rate for the second individual coolingmedia used to address the second heat feature in the datacenter. In atleast one embodiment, the instruction outputs of the processor are pinsof a connector bus or balls of a ball grid array, which enablescommunication of the outputs in the form of signals carryinginstructions. The at least one processor is part of a distributed or acentralized control system to control the one or more flow controllers.

FIG. 2B is a block diagram illustrating an evaporative cooling subsystem220 benefiting from a repurposable refrigerant cooling subsystem,according to at least one embodiment. In at least one embodiment, theevaporative cooling subsystem 220 takes in supply air 222A, from thedatacenter (or other appropriate source) using supply fan 224A thatfunctions to suction the supply air. In at least one embodiment, thesupply air then passes through filters 226 to an indirect heat exchanger232. In at least one embodiment, the indirect heat exchanger has coilsor perforated plates 232A. Scavenger air 240A is suctioned through theindirect heat exchanger 232 by an action of scavenger fan 224B thatblows the scavenger air as exhaust air 240B from a provided facility ofthe evaporative cooling subsystem 220.

In at least one embodiment, the evaporative cooling subsystem 220includes sump 234 having an appropriate evaporative media, such asfacility water. A sump pump 236A is provided to pump the evaporativemedia through line 236B for a spraying system 236C that sprays theevaporative media 230 on to the coils or the perforated plates 232A (orthrough the indirect heat exchanger) while the scavenger air 240A ispassing through and while the supply air 222A is passing through. Thesupply air 222A is mixed with the scavenger air 240A, in at least oneembodiment. In at least one embodiment, the supply air 222A is separatefrom the scavenger air 240A, and flows through the coils or theperforated plates 232A, while picking up moisture, while the scavengerair 240A separately acts to cool the coils or the perforated plates232A, which in turn cools the supply air 222A as it passes through.

In at least one embodiment, the supply air 222A has some moisture as aresult of the spray of the evaporative media 230, but is at least partlyconditioned for cooling the datacenter per at least one heat feature orcooling requirement of the datacenter. The air 238B exiting the indirectheat exchanger 232 may be provided to the datacenter as blown air 222B.In at least one embodiment, the evaporative cooling subsystem 220provides blown air for the datacenter according to the first heatfeature in the datacenter. In addition, the evaporative coolingsubsystem 220 may include components 228, 242, 246 that work with arepurposable refrigerant cooling subsystem to control moisture of theblown air in a first configuration of the repurposable refrigerantcooling subsystem. In at least one embodiment, the repurposablerefrigerant cooling subsystem administers refrigerant via a condenser228 that cools the refrigerant and that passes the refrigerant to theevaporator coil 242. In at least one embodiment, the refrigerant exitingthe evaporator coil 242 is compressed by direct cooling compressor 245and returned to the repurposable refrigerant cooling subsystem. In atleast one embodiment, the compressed refrigerant is in vapor state andis fed back to the condenser 228 to exchange heat by the scavenger air240A flowing through the indirect heat exchanger 232. The repurposablerefrigerant cooling subsystem is separately able to use the refrigerantto cool the datacenter in a second configuration according to a secondheat feature in the datacenter.

In at least one embodiment, the refrigerant is at a first flow rate whenit flows through the evaporator coil, which ensures that the evaporatorcoil 242 performs a dual function. In at least a first configuration,the first flow rate of refrigerant through the evaporator coil 242 is toensure that the coil is sufficiently cool to cause moisture in the air238B to condense on the evaporator coil 242. As such, moisture isremoved from the air 238B in the first configuration. In at least asecond configuration, a second flow rate of the refrigerant is so thatit can address cooling requirements inside the datacenter. In at leastone embodiment, the air 238B is blown air 222B returned to thedatacenter for cooling the datacenter according to a first heat featureor cooling requirement. This may be the case when the air 238B has adetermined threshold of moisture that is within a limit adapted for thedatacenter (such as 55-60%). In at least one embodiment, the air 238B isadjusted for moisture content by the refrigerant through the evaporatorcoil 242, so that the air 238B has first moisture content, but the blownair 222B that is returned to the datacenter for cooling the datacenteraccording to the first heat feature or cooling requirement has a secondmoisture content that is lower than the first moisture content. This isso as to not affect the determined threshold of moisture that is withinthe limit adapted for the datacenter.

FIG. 2C is a block diagram illustrating datacenter-level features of acooling system 250 incorporating a evaporating cooling subsystem 278 anda repurposable refrigerant cooling subsystem 282 within a multiple modecooling subsystem 256, according to at least one embodiment. The coolingsystem 250 is for a datacenter 252. The repurposable refrigerant coolingsubsystem 282 includes support for humidity controls of an evaporativecooling system 278. The multiple mode cooling subsystem 256 may furtherinclude a liquid-based cooling subsystem 280 and/or a dielectric coolingsubsystem 284. The repurposable refrigerant cooling subsystem 282 mayhave overlapping components with the evaporative cooling subsystem, andis therefore, adapted for two different configuration as discussed withreference to FIGS. 2A, 2B. Further, the multiple mode cooling subsystem256 may be rated for different cooling capacities as each of thesubsystems 278-284 may have a different rated cooling capacity. In atleast one embodiment, the evaporative cooling system 278 is rated forbetween 10 kilowatt (KW or kW) to 30 KW of generated heat, theliquid-based cooling subsystem 280 is rated for between 30 KW to 60 KW,the repurposable refrigerant cooling subsystem 282 is rated for between60 KW to 100 KW (in the cooling configuration, different than thehumidity control configuration), and the dielectric cooling subsystem282 is rated for between 100 KW to 450 KW of generated heat.

In at least one embodiment, the generated heat from is one or morecomputing component in servers of the racks of the datacenter. Therating is a feature of a cooling subsystem that can address (by coolingor by dissipating, in at least one embodiment) the generated heat whenit is at a range addressable by the cooling subsystem. Different coolingmedia are able to address the generated heat differently according tothe rating noted for the different cooling media. In at least oneembodiment, for the repurposable refrigerant cooling subsystem 282, thecooling media refrigerant; and for the evaporative cooling subsystem274, the cooling media is blown air. The generated heat from the one ormore computing components, servers, or racks is different than theratings afforded to the cooling subsystems for their individualcapability to address generated heat, and is referred to a heat featureor a cooling requirement that must be addressed by the different coolingcapacities of the cooling subsystems.

In at least one embodiment, as the multiple mode cooling subsystem 256is rated for different cooling capacities because of at least thedifferent cooling media from the different cooling subsystems 278-284therein, the multiple mode cooling subsystem 256 is adjustable todifferent cooling requirements of the datacenter 252 within a minimumand a maximum of the different cooling capacities offered by thedifferent cooling subsystems 278-284. In at least one embodiment, thedifferent cooling media include air or gas (such as blown air), one ormore different types of coolant (including water and glycol-basedcoolants), and one or more different types of refrigerant (includingnon-dielectric and dielectric refrigerants). In at least one embodiment,the multiple mode cooling subsystem 256 has a form factor for one ormore racks 254 of the datacenter. In at least one embodiment, themultiple mode cooling subsystem 256 is in the form factor of one or moreracks and is installed adjacent to one or more server racks 254 toaddress the different cooling requirements of components and servers ofthe one or more server racks 254.

In at least one embodiment, the one or more server racks 254 hasdifferent cooling requirements based in part on different heat generatedby components of server boxes or trays 270-276 within the one or moreserver racks 254. In at least one embodiment, multiple server boxes 270have nominal heat generated, which corresponds to about 10 KW to 30 KWof generated heat, and for which the evaporative cooling subsystem 278is activated to provide evaporative cooling as illustrated by thedirectional arrows from subsystem 278. The evaporative cooling is inreference to blown air 278B provided from the evaporative coolingsubsystem 278. The evaporative cooling subsystem 278 may sharecomponents with the repurposable refrigerant cooling subsystem 282 sothat the blown air 278B provided to the sever boxes 270 is a cooled andhumidity-controlled air. The blown air 278B may be cooled by an indirectheat exchanger within the evaporative cooling subsystem 278, asdescribed with respect to FIG. 2B. The heat exchanger within theevaporative cooling subsystem 278 is part of an evaporative-basedsecondary cooling loop 292 that exchanges heat with a coolant-basedprimary cooling loop 288 associated with a CDU 286. In at least oneembodiment, the coolant is facility water that is transferred to thesump, as illustrated and discussed in FIG. 2B. In at least oneembodiment, a chilling facility formed of at least a chiller unit 258, acooling tower 262, and a pump 260 supports the coolant-based primarycooling loop 288.

In at least one embodiment, the cooling system of the disclosureincludes a refrigerant-based heat transfer subsystem to enable one ormore of the repurposable refrigerant cooling subsystem 282 and thedielectric cooling subsystem 284; and includes a liquid-based heattransfer subsystem to enable one or more of the evaporative coolingsystem 278 and the liquid-based cooling subsystem 280. In at least oneembodiment, the refrigerant-based heat transfer subsystem includes atleast pipes 278A to support the refrigerant flow to the differentcooling subsystems, and may include one or more evaporative coils,compressors, condensers, and expansion valves (some of wish arediscussed and illustrated in FIG. 2B) to support circulation of therefrigerant for the different cooling subsystems. In at least oneembodiment, a dielectric refrigerant is used to enable therefrigerant-based cooling subsystem 282 and the dielectric coolingsubsystem 284. This enables reduction in components required for thecooling system of the present disclosure, but supports a wide range ofcooling capabilities to address the different cooling requirements madeof the system. Furthermore, in at least one embodiment, a coolant-basedsecondary cooling loop 294 supports the liquid-based cooling subsystem280.

In at least one embodiment, as a result of at least the aboveconfiguration, even though server trays 270 are air-cooled byevaporative cooling subsystem 278, server tray 272 may be liquid cooled(using a coolant, in at least one embodiment), via liquid-based coolingsubsystem 280; while server tray 279 may be immersion cooled (using adielectric refrigerant, in at least one embodiment), via thedielectric-based cooling subsystem 284; and server tray 276 may berefrigerant cooled (using the same dielectric refrigerant or a separaterefrigerant, in at least one embodiment). In at least one embodiment,the refrigerant and the dielectric cooling subsystems 282, 284 isdirectly coupled to at least one condensing unit 264 to address thecooling requirement of the refrigerant via a refrigerant cooling loop290.

In at least one embodiment, a coolant distributor, such as a coolantdistribution unit 286, provides coolant for the evaporative coolingsubsystem 278 and the liquid-based cooling subsystem 280. In at leastone embodiment, for the liquid-based cooling subsystem 280, a primarycoolant cools a secondary coolant that circulates via the coolant-basedsecondary cooling loop 294 to cool components within the coolant-cooledserver tray 272. Although server tray 272 is referred to as acoolant-cooled server tray, it is appreciated that the coolant lines areavailable to all server trays 270-276 of rack 254 and to all racks ofthe datacenter 252 via at least row manifold 266, and respectiveinlet-side rack manifold(s) 296A and outlet side rack manifold(s) 296B.

In at least one embodiment, the racks 254, 256 (including a rack orracks 256 forming the cooling subsystem) are on a raised platform orfloor 268, and the row manifold 266 may be a collection of manifoldswithin it to reach all the server trays or select server trays in theavailable racks of the datacenter. Similarly, in at least oneembodiment, for the refrigerant and immersion-based cooling subsystems282, 284, one or more of a dielectric refrigerant or a regularrefrigerant circulates via a refrigerant-based secondary cooling loop294 (that may extend from the primary cooling loop 290) to coolcomponents within the refrigerant and/or the immersion-cooled servertrays 279, 276. In at least one embodiment, although server trays 279,276 are referred to as refrigerant and/or the immersion-cooled servertrays 279, 276, it is appreciated that the refrigerant (or dielectricrefrigerant) lines are available to all server trays 270-276 of rack 254and to all racks of the datacenter 252 via at least row manifold 266,and respective inlet-side rack manifold(s) 296A and outlet side rackmanifold(s) 296B. The row manifold and the rack manifolds areillustrated as single lines, but may be a collection of manifolds tocarry coolants, refrigerants, and dielectric refrigerants distinctly, toreach all the server trays or select server trays in the available racksof the datacenter via one or more associated flow controllers. In thismanner, in at least one embodiment, all the server trays may be able toseparately request (or indicate a requirement for) a desired level ofcooling from the multiple mode cooling subsystem 256.

In at least one embodiment, a first individual cooling subsystem, suchas the coolant-based cooling subsystem 280, of the multiple mode coolingsubsystem 256 has associated first looping components (such ascoolant-based secondary cooling loop in row manifold 266 and in rackmanifolds 296A, 296B) to guide a first cooling media (such as thecoolant) to at least one first rack 254 of the datacenter 252 accordingto a first heat feature or a cooling requirement of the at least onefirst rack 254. In at least one embodiment, the multiple mode coolingsubsystem 256 has associated inlet and outlet lines 298A to circulateone or more cooling media in and out of the appropriate manifolds of therow manifold 266. The first heat feature or the cooling requirement maybe a temperature sensed by a sensor associated with the at least onefirst rack 254. In at least one embodiment, concurrently with thecoolant-based cooling subsystem 280 cooling the at least one first rack254 (or a server therein), the refrigerant-based cooling subsystem 282of the multiple mode cooling subsystem 256 has associated second loopingcomponents (such as a refrigerant-based second cooling loop in rowmanifold 266 and in rack manifolds 296A, 296B) to guide a second coolingmedia (such as a refrigerant or dielectric refrigerant) to at least onesecond rack of the datacenter according to a second heat feature or asecond cooling requirement of the at least one second rack. This type ofdifferent cooling media addressing different requiring racks is alsopossible to address different requiring servers within the same rack asalready illustrated by rack 254 in FIG. 2C.

In at least one embodiment, the first heat feature or the first coolingrequirement is within a first threshold of the minimum of the differentcooling capacities. In at least one embodiment, the minimum of thedifferent cooling capacities is provided by the evaporative coolingsubsystem 278. In at least one embodiment, the second heat feature orthe second cooling requirement is within a second threshold of themaximum of the different cooling capacities. In at least one embodiment,the maximum of the different cooling capacities is provided by thedielectric-based cooling subsystem 284 that allows full immersion of thecomputing components in a dielectric media to cool the computingcomponents by direct convection cooling as opposed to air, coolant, andrefrigerant based media that work by radiative and/or conductivecooling. In at least one embodiment, cold plates associated with thecomputing components draw heat from the computing components andtransfers the heat to the coolant or refrigerant when these types ofcooling media are used. In at least one embodiment, the cold platestransfers the heat to air flowing through the cold plates (either havingassociated fins or directly via the cold plates) when the cooling mediais air.

In at least one embodiment, inlet and outlet lines 298B, to rackmanifold 296A, 296B of the one or more racks 254, enable individualservers 270-276 associated with the rack manifold(s) 296A, 296B toreceive different cooling media from individual cooling subsystems280-284 of the multiple mode cooling subsystem 256. In at least oneembodiment, one or more servers 270-279 have or support two or more ofan air distribution unit, a coolant distribution unit, and a refrigerantdistribution unit to enable different cooling media to address thedifferent cooling requirements of the one or more servers. FIGS. 3A-4Baddresses server-level features of such servers, but briefly, acoolant-based heat exchanger (a coolant-based cold plate orcoolant-lines supported cold plate, in at least one embodiment) may beused as a coolant distribution unit for the server 272; arefrigerant-based heat exchanger (refrigerant-based cold plate orrefrigerant-lines supported cold plate, in at least one embodiment) maybe used as a refrigerant distribution unit for the server 279, and oneor more fans may be used as an air distribution unit for the servers270.

In at least one embodiment, one or more sensors 278D are provided tomonitor air quality (or an environment) within the datacenter 252,within a rack 254, or within a server 270. In at least one embodiment,the one or more sensors 278D is provided close to a return air manifoldor a supply air manifold that is associated with the evaporative coolingsubsystem 278. In at least one embodiment, the one or more sensors 278Dmeasure temperature and humidity of the air within the datacenter 252,within the rack 254, or within the server 270. In at least oneembodiment, the one or more sensors provide input to a distributed or acentralized control system, as discussed throughout this disclosure.

In at least one embodiment, the distributed or the centralized controlsystem includes at least one processor and memory including instructionsto cause the distributed or the centralized control system to react tothe input. In at least one embodiment, when the input is indicative of afirst temperature reflecting a first heat feature or a first coolingrequirement, the distributed or the centralized control system is ableto determine that blown air from the evaporative cooling subsystem 278is the best cooling solutions. The blown air 278B is provided. Thedistributed or the centralized control system is still in communicationwith the sensors 278D to determine moisture content in the blown air278B. The sensors 278D indicate moisture presence (which may include anamount of the moisture) in the return air, which forms the blown air278D, from the evaporative cooling subsystem. The distributed or thecentralized control system is able to determine an effect of themoisture on the air quality based in part on inputs form the sensors278D, as to humidity. In at least one embodiment, an input of themoisture sensed at two different time intervals may be a basis todetermine the amount of the moisture in the return air. In at least oneembodiment, the distributed or the centralized control system is able touse information of the effect of the moisture on the air quality todetermine which, if any, flow controllers to open or close.

In at least one embodiment, the flow controllers to be opened or closedare associated with both the evaporative cooling subsystem 278 and therepurposable refrigerant cooling subsystem 282. In at least oneembodiment, first flow controllers may be modified to control ormaintain a flow rate of a first cooling media, such as the blown air278B from the evaporative cooling subsystem. In at least one embodiment,second flow controllers may be modified to control a flow rate of asecond media, such as refrigerant on lines 278A from the repurposablerefrigerant cooling subsystem 282, to be used in the evaporative coolingsubsystem 278 for controlling the moisture that may have caused theeffect on the air quality. In at least one embodiment, components of therepurposable refrigerant cooling subsystem 282 may be shared with theevaporative cooling subsystem 278 so that the refrigerant is providedinternally, from a condenser to an evaporator coil. The moisture, ifany, in the return air is condensed on the evaporator coil and removedprior to the return air entering the datacenter 252, the rack 254, orthe server 270 as the blown air. This reflects at least a firstconfiguration for the repurposable refrigerant cooling subsystem 282.

In at least one embodiment, when the sensors 278D provide input that isindicative, to the distributed or the centralized control system, thatthe temperature within the datacenter 252, the rack 254, or the server270 is increasing or not responding sufficiently to the blown airprovided from the evaporative cooling subsystem 278, the distributed orthe centralized control system is able to cause the first flowcontrollers to seize activation, and is able to cause the second flowcontrollers or third flow controllers associated with the repurposablerefrigerant cooling subsystem 282 to provide refrigerant at a third flowrate that may be higher than the second flow rate. As the refrigerant iscaused to flow to a rack or a server via cooling lines 298A, a rowmanifold 266, rack lines 298B, and rack manifolds 26A, 296B, the flowcontrollers may be different than used for the line 278A to theevaporative cooling subsystem. The refrigerant is caused to flow viarefrigerant lines of the row manifolds 298A to the datacenter 252, therack 254, or the server 270 to address the increasing temperature thatalso reflects a second heat feature or second cooling requirement. In atleast one embodiment, the first cooling media may be maintained whilethe second cooling media is introduced. The first cooling media may bedeactivated after a period of time with the second cooling mediacontinuing to flow. In at least one embodiment, the second cooling mediamay be caused to flow within an evaporator coil in response to thesecond heat feature to perform a dual purposes of removing moisture inthe datacenter 252, the rack 254, or the server 270, and for addressingthe second heat feature by flowing through requisite cooling components,such as a cold plate associated with a component requiring cooling. Thisreflects at least a second configuration for the repurposablerefrigerant cooling subsystem 282. In at least one embodiment, a flowvolume of the refrigerant may be higher in the second configuration thanin the first configuration because they address different requirementsof the repurposable refrigerant cooling subsystem 282.

In at least one embodiment, as a result of the above features, therepurposable refrigerant cooling subsystem 282 is caused to function inthe second configuration when the evaporative cooling subsystem 278 isineffective in achieving a reduction in heat generated in at least oneserver or rack of the datacenter. Further, in at least one embodiment,the evaporative cooling subsystem 278 and the repurposable refrigerantcooling subsystem 282 provide a determined range of cooling capacitiesto support mobility of the datacenter by at least the differentutilities offered by the repurposable refrigerant cooling subsystem 282.

In at least one embodiment, the evaporative cooling subsystem 278 hasassociated first looping components, such as supply air manifold 222Dand return air manifold 222C in FIG. 2C, to guide a first cooling mediato at least one first rack of the datacenter according to a first heatfeature or a first cooling requirement of the at least one first rack.In at least one embodiment, the repurposable refrigerant coolingsubsystem 282 has associated second looping components, such as withinsupply and return lines 298A, 298B, and within manifolds 266, 296A,296B, to guide a second cooling media to the at least one first rack ofthe datacenter according to a second heat feature or a second coolingrequirement of the at least one first rack. In at least one embodiment,the first heat feature or the first cooling requirement is within afirst threshold of the different cooling capacities offered by themultiple mode cooling subsystem 256 having at least the evaporative andthe repurposable refrigerant cooling subsystems 278, 282. In at leastone embodiment, the second heat feature or the second coolingrequirement is within a second threshold of the different coolingcapacities. In at least one embodiment, the first threshold is a valuethat is between 8 to 9 KW as the evaporative cooling subsystem canaddress between 10 KW to 30 KW of generated heat. In at least oneembodiment, the second threshold is between 31 to 59 KW, as therepurposable refrigerant cooling subsystem 282 is rated for between 60KW to 100 KW of generated heat. In at least one embodiment, aliquid-based cooling subsystem 280 can meet the intermediaterequirements between the evaporative and the repurposable refrigerantcooling subsystems. In at least one embodiment, the evaporative coolingsubsystem is kept active along with the liquid-based cooling subsystem,till the repurposable refrigerant cooling subsystem is activated.

In at least one embodiment, at least one processor for a cooling systemis disclosed, such as processor 1400 in FIG. 14 that may use the neurons1502 and components thereof implemented using circuitry or logic,including one or more arithmetic logic units (ALUs) as described in FIG.15. The at least one logic unit is adapted to control flow controllersassociated with the repurposable refrigerant cooling subsystem 282 andthe evaporative cooling subsystem 278. The control is based in part onthe first heat feature in the datacenter and based in part on the secondheat feature in the datacenter. The evaporative cooling subsystem 278works with the repurposable refrigerant cooling subsystem to controlmoisture of the blown air 278B from the evaporative cooling subsystem ina first configuration of the repurposable refrigerant cooling subsystem,as discussed throughout this disclosure. The repurposable refrigerantcooling subsystem 282 is adapted to cool the datacenter in a secondconfiguration of the repurposable refrigerant cooling subsystem. The atleast one processor may provide outputs having instructions to the flowcontrollers to enable the first and the second configurations, in atleast one embodiment. The blown air 278B and the cooling of thedatacenter is according to the first heat feature and the second heatfeature respectively.

In at least one embodiment, a learning subsystem in the ALUs is appliedto evaluate temperatures within servers or one or more racks in thedatacenter with flow rates of different cooling media. The learningsubsystem provides the outputs that are associated with at least twotemperatures to facilitate movement of two cooling media from thecooling system by controlling the flow controllers using theinstructions. The flow controllers are associated with the two coolingmedia to address the first heat feature and the second heat feature.

In at least one embodiment, the learning subsystem executes a machinelearning model. The machine learning model process the temperaturesusing multiple neuron levels of the machine learning model having thetemperatures and having prior associated flow rates for the two coolingmedia. In at least one embodiment, the learning subsystem provides theoutputs associated with flow rates for the two cooling media to the flowcontrollers. The outputs are provided after an evaluation of the priorassociated flow rates of individual cooling media of the two coolingmedia and prior associated temperatures.

In at least one embodiment, the at least one processor includesinstruction outputs, such as pins and balls to communicate the outputsassociated with the flow controllers. In at least one embodiment a pumpis adapted to receive the outputs from the at least one processor and isadapted to adjust an internal motor or a valve to change a flow throughthe pump. The outputs enable a first flow rate of the first individualcooling media used to address the first heat feature in the datacenter,while maintaining a second flow rate of a second individual coolingmedia to control humidity of the first individual cooling media. Theoutputs may concurrently deactivate the first individual cooling mediaand may enable a third flow rate for the second individual cooling mediaused to address the second heat feature in the datacenter.

In at least one embodiment, the at least one logic unit is adapted toreceive temperature values from temperature sensors and humidity valuesfrom humidity sensors, which are both implemented in sensors 278D, in atleast one embodiment. The temperature sensors and the humidity sensorsare, therefore, associated with the servers or the one or more racks,and adapted to facilitate movement of a first individual cooling mediaand a second individual cooling media concurrently to address the firstheat feature of the data center, and individually to address the secondheat feature of the datacenter. In at least one embodiment, by itscommunication with the at least one processor, within the centralized orthe distributed control system, the sensors 278D are adapted tofacilitate the movement of the first and the second individual coolingmedia.

In at least one embodiment, the at least one processor has at least onelogic unit to train one or more neural network having hidden layers ofneurons. The training is for the one or more neural network to evaluatetemperatures, flow rates, and humidity that are associated with arepurposable refrigerant cooling subsystem 282 and an evaporativecooling subsystem 278 based in part on a first heat feature in thedatacenter 252 and based in part on a second heat feature in thedatacenter 252. The evaporative cooling subsystem 278 works with therepurposable refrigerant cooling subsystem 282 to control moisture ofblown air 278B from the evaporative cooling subsystem 278 in a firstconfiguration of the repurposable refrigerant cooling subsystem. Therepurposable refrigerant cooling subsystem 282 acts to cool thedatacenter 252 in a second configuration of the repurposable refrigerantcooling subsystem 282. The blown air 278B and the cooling of thedatacenter is according to the first heat feature and the second heatfeature respectively. The repurposable refrigerant cooling subsystem 282provides refrigerant directly to a rack, a server, or a cold plate of acomponent to provide the cooling in the second configuration.

In at least one embodiment, the at least one logic unit outputs at leastone instruction associated with at least two temperatures to facilitatemovement of the at least two cooling media by controlling flowcontrollers associated with the at least two cooling media. The flowcontrollers facilitate flowing of the two cooling media concurrently inthe first configuration of the repurposable refrigerant coolingsubsystem and facilitates flowing of one of the two cooling media alonein the second configuration of the repurposable refrigerant coolingsubsystem.

In at least one embodiment, the at least one processors has at least oneinstruction output to communicate the output associated with flowcontrollers. In at least one embodiment, the output communicated to theflow controllers enables a first flow rate of a first individual coolingmedia used to address the first heat feature in the datacenter, whilemaintaining a second flow rate of a second individual cooling media tocontrol humidity of the first individual cooling media. In at least oneembodiment, the output communicated to the flow controllers deactivatesthe first individual cooling media and enables a third flow rate for thesecond individual cooling media used to address the second heat featurein the datacenter.

FIG. 3A is a block diagram illustrating rack-level 300A and server-level300B features of a cooling system incorporating an evaporative coolingsubsystem in the multiple mode cooling subsystem, according to at leastone embodiment. In at least one embodiment, the evaporative coolingsubsystem is able to address an entire rack 302 by blowing cold air 310through the entire rack 302 and cycling hot air 312 received from theopposite end of the rack 302. In at least one embodiment, one or morefans are provided in a conduit or an air manifold 318 to circulate thecold air 314, which may be the cold air 310 at the entry of the rack302, from the evaporative cooling subsystem to the individual server304. Hot air 316 flows out from an opposite end of the server 304 tojoin the hot air 312 exiting the rack 302. As such, server 304 is one ormore of servers 306 in rack 302, and has one or more componentsassociated with cold plates 308A-D and requiring a nominal level ofcooling. The cold plates 308A-D may have associated fins to enable moreheat dissipation for beneficial use of the air cooling media.

In at least one embodiment, the components associated with cold plates308A-D are computing components generating between 50 KW to 60 KW ofheat and sensors associated with the one or more servers, or the rackentirely, may indicate the temperate (representing a coolingrequirement) to the multiple mode cooling subsystem, which can thendetermine the specific one of the cooling subsystems 278-284 to activatefor the one or more servers or the rack entirely. In at least oneembodiment, the indication of the temperature may be sent to the coolingsubsystems 278-284 and the cooling subsystem designated to address aspecific temperature corresponding to KW heat generated (for thespecific temperature) activates itself to address the server or theracks requiring the cooling. This represents a distributed controlsystem, where control of activation or deactivation is shared byprocessors of the different subsystems 278-284. In at least oneembodiment, a centralized control system can be enabled by one or moreprocessors of a server 306 in the server rack 302. The one or moreprocessors is adapted to perform the controlling requirements for themultiple mode cooling system, including by using artificial intelligenceand/or machine learning to train a learning subsystem to recognizetemperature indications and to activate or deactivate appropriatecooling subsystems 278-284. The distributed control system may be alsoadapted to use artificial intelligence and/or machine learning to traina learning subsystem, albeit by a distributed neural network, forinstance.

FIG. 3B is a block diagram illustrating rack-level 330A and server-level330B features of a cooling system incorporating a liquid-based coolingsubsystem of the multiple mode cooling subsystem, according to at leastone embodiment. In at least one embodiment, the liquid-based coolingsubsystem is a coolant-based cooling subsystem. The rack-level features330A supporting or incorporating the liquid-based cooling subsystemincludes a rack 332 having inlet and outlet coolant-based manifolds358A, 358B, having associated therewith inlet and outlet couplers oradapters 354, 356, and having associated therewith individual flowcontrollers 362 (two are referenced in the figure). The inlet and outletcouplers or adapters 354, 356 receive, direct, send coolant 352 throughone or more requesting (or indicating) servers 336; 304. In at least oneembodiment, all the servers 336 in the rack 332 may be addressed bytheir individual requests (or indications) for the same coolingrequirement, or by a request (or indication) from the rack 332 on behalfof its servers 336, which can be addressed by the liquid-based coolingsubsystem. Heated coolant 360 exits the rack via adapter or coupler 356.The individual flow controllers 362 may be pumps that are controlled(activated, deactivated, or maintained in flow position) via inputs fromthe distributed or centralized control system, in at least oneembodiment. Further, in at least one embodiment, when the rack 332 makesits request (or providing its indication) for a cooling requirement, allthe flow controllers 362 may be simultaneously controlled to address thecooling requirement, as if the cooling requirement is representative ofcooling requirements of all the servers 336 at the same time.

In at least one embodiment, the request or the indication is made from asensor associated with the rack or with the servers. There may be one ormore sensors in different locations of the rack and in differentlocations of the server. The sensors, in a distributed configuration,may send individual requests or indications of a temperature, in atleast one embodiment. In at least one embodiment, the sensors work in acentralized configuration and are adapted to send its requests orindications to one sensor or to a centralized control system thataggregates the requests or indications to determine a singular action.In at least one embodiment, the singular action may be to send one ormore cooling media from the multiple mode cooling subsystem to addressthe requirements or indications associated with the singular action. Inat least one embodiment, this enables the present cooling system to sendcooling media associated with a higher cooling capability than theaggregate cooling requirement of some of the servers in the racks as theaggregated requested or indications may include at least some coolingrequirements that are higher than the aggregate cooling requirement.

In at least one embodiment, a mode statistical measure is used as anaggregate to indicate the number of servers having higher coolingrequirement, which may be addressed instead of a pure average of thecooling requirements of the servers. As such, it is appreciated that asubsystem offering higher cooling capability (such as refrigerant-basedcooling subsystem) may be activated when at least 50% of the servers ofa rack have higher cooling requirements. In at least one embodiment, asubsystem offering lower cooling capability (such as coolant-basedcooling subsystem) may be activated when at most 50% of the servers ofthe rack have higher cooling requirements. However, when a number of theservers of the rack start requesting or indicating higher coolingrequirements after the coolant-based cooling subsystem is activated forthe rack, then a higher cooling capability may be provided to addressthe shortfall. In at least one embodiment, the change of the specificcooling subsystem may occur to higher or lower capability cooling basedin part on an amount of time elapsed since a first request or indicationoccurred and when the first request or indicate persists or shows anelevation of an associated temperature.

In at least one embodiment, the servers 336 have individual server-levelmanifolds, such as manifold 338 in server 334 (also referred to asserver tray or server box). The manifold 338 of server 334 includes oneor more server-level flow controllers 340A, B for addressing individualcomponent(s) requirements for cooling. The server manifold 338 receivescoolant from rack manifold 358A (via adapter or coupler 348) and,depending on requirements of the computing components associated withcold plates 342A-D, is caused to distribute the coolant (or anassociated server-level coolant exchanging heat with the rack levelcoolant) to the components via the associated cold plates 342A-D. In atleast one embodiment, the distributed or centralized control systemassociated with the multiple mode cooling subsystem is able to controlthe rack level flow controllers 362 and the server level flowcontrollers 340A, B to facilitate movement of coolant from theliquid-based cooling subsystem. As such, in at least one embodiment, asensor on a first side of the server 334 having components associatedwith a first set of cold plates 342A, B may indicate or request highercooling requirements than a sensor on a second side of the server 334having components associated with a second set of cold plates 342C, D.Then the coolant, if a coolant is sent as cooling media, is enabled toflow, via flow controller 340A and inlet line 344A, through the firstset of cold plates 342A, B, out via outlet line 346A, and back to theserver manifold 338. The heated coolant may exit the server manifold viaadapter or coupler 350.

In at least one embodiment, the server manifold 338 as well as the rowmanifold(s) 358A, B include multiple sub-manifolds or lines andassociated couplers to support the other cooling media, including therefrigerant (or the dielectric refrigerant), but is illustrated and isdiscussed here for the coolant (liquid-based media) portion as relevantto this feature. In at least one embodiment, as the sensor on the secondside of the server 334 having components associated with the second setof cold plates 342C, D did not indicate or request higher coolingrequirements than the sensor on the first side of the server 334 havingcomponents associated with the first set of cold plates 342A, B, theflow controller 340B remains closed or unchanged (if previously openedfor a different cooling media). There will be no change to or no coolingmedia in inlet line 344B, the second set of cold plates 342C, D, and theoutlet line 346B to the server manifold 338.

FIG. 4A is a block diagram illustrating rack-level 400A and server-level400B features of a cooling system incorporating a repurposablerefrigerant cooling subsystem in the multiple mode cooling subsystem,according to at least one embodiment. The rack-level features 400Asupporting or incorporating the repurposable refrigerant coolingsubsystem includes a rack 402 having inlet and outlet refrigerantmanifolds 428A, 428B, having associated therewith inlet and outletcouplers or adapters 424, 426, and having associated therewithindividual flow controllers 432 (two are referenced in the figure). Theinlet and outlet couplers or adapters 424, 426 receive, direct, sendrefrigerant 422 through one or more requesting (or indicating) servers406; 404. In at least one embodiment, all the servers 406 in the rack402 may be addressed by their individual requests (or indications) forthe same cooling requirement, or by a request (or indication) from therack 402 on behalf of its servers 406, which can be addressed by therepurposable refrigerant cooling subsystem in the second configurationfor the repurposable refrigerant cooling subsystem. Liquid phaserefrigerant may enter the rack manifold via adapter or coupler 424 andvapor phase refrigerant 430 exits the rack via adapter or coupler 426.The individual flow controllers 432 may be pumps that are controlled(activated, deactivated, or maintained in flow position) via inputs fromthe distributed or centralized control system as discussed in referenceto FIGS. 2A-3B, in at least one embodiment. Further, in at least oneembodiment, when the rack 402 makes its request (or providing itsindication) for a cooling requirement, all the flow controllers 432 maybe simultaneously controlled to address cooling requirement as if thecooling requirement is representative of cooling requirements of all theservers 406, at the same time. In at least one embodiment, the requestor the indication comes from a sensor associated with the rack thatsenses temperature and/or humidity within the rack.

In at least one embodiment, the request or the indication is made from asensor associated with the rack or with the servers. As in theembodiment of FIG. 3B, there may be one or more sensors in differentlocations of the rack and in different locations of the server. Thesensors, in a distributed configuration, may send individual requests orindications of a temperature, in at least one embodiment. In at leastone embodiment, the sensors work in a centralized configuration and areadapted to send its requests or indications to one sensor or to acentralized control system that aggregates the requests or indicationsto determine a singular action. In at least one embodiment, the singularaction may be to send one or more cooling media from the repurposablerefrigerant cooling subsystem to address the requirements or indicationsassociated with the singular action. In at least one embodiment, thisenables the present cooling system to send cooling media associated witha higher cooling capability than the aggregate cooling requirement ofsome of the servers in the racks as the aggregated requested orindications may include at least some cooling requirements that arehigher than the aggregate cooling requirement.

In at least one embodiment, a mode statistical measure is used as anaggregate to indicate the number of servers having higher coolingrequirement, which may be addressed instead of a pure average of thecooling requirements of the servers. As such, it is appreciated that asubsystem offering higher cooling capability (such as immersion-basedcooling subsystem) may be activated when at least 50% of the servers ofa rack have higher cooling requirements. However, the server requestinghigher cooling capability must also support the higher coolingcapability. In at least one embodiment, a server supporting immersioncooling must have a sealed space in which to allow a dielectric coolingmedia, such as a dielectric refrigerant to contact components directly.In at least one embodiment, a subsystem offering lower coolingcapability (such as the refrigerant or a coolant-based coolingsubsystem) may be activated when at most 50% of the servers of the rackhave higher cooling requirements. However, when a number of the serversof the rack start requesting or indicating higher cooling requirementsafter the refrigerant or a coolant-based cooling subsystem is activatedfor the rack, then a higher cooling capability may be provided toaddress the shortfall. In at least one embodiment, the change of thespecific cooling subsystem may occur to higher or lower capabilitycooling based in part on an amount of time elapsed since a first requestor indication occurred and when the first request or indicate persistsor shows an elevation of an associated temperature.

In at least one embodiment, the servers 406 have individual server-levelmanifolds, such as manifold 408 in server 404. The manifold 408 ofserver 404 includes one or more server-level flow controllers 410A, Bfor addressing individual component(s) requirements for cooling. Theserver manifold 408 receives liquid phase refrigerant from rack manifold428A (via adapter or coupler 418) and, depending on requirements of thecomputing components associated with cold plates 412A-D, is caused todistribute the refrigerant to the components via the associated coldplates 412A-D. In at least one embodiment, the distributed orcentralized control system associated with the repurposable refrigerantcooling subsystem is able to control the rack level flow controllers 432and the server level flow controllers 410A, B to facilitate movement ofrefrigerant from the repurposable refrigerant cooling subsystem. Assuch, in at least one embodiment, a sensor on a first side of the server404 having components associated with a first set of cold plates 412A, Bmay indicate or request higher cooling requirements than a sensor on asecond side of the server 404 having components associated with a secondset of cold plates 4122C, D. Then the refrigerant, if a refrigerant issent as cooling media, is enabled to flow, via flow controller 410A andinlet line 414A, through the first set of cold plates 412A, B, out viaoutlet line 416A, and back to the server manifold 408. The vapor phaserefrigerant may exit the server manifold via adapter or coupler 420.

In at least one embodiment, the server manifold 404 as well as the rowmanifold(s) 428A, B include multiple sub-manifolds or lines andassociated couplers to support the other cooling media, including thecoolant (or the dielectric refrigerant), but is illustrated and isdiscussed here for the refrigerant (repurposable refrigerant media)portion as relevant to this feature. In at least one embodiment, as thesensor on the second side of the server 404 having components associatedwith the second set of cold plates 412C, D did not indicate or requesthigher cooling requirements than the sensor on the first side of theserver 404 having components associated with the first set of coldplates 412A, B, the flow controller 410B remains closed or unchanged (ifpreviously opened for a different cooling media). There will be nochange to or no cooling media in inlet line 414B, the second set of coldplates 412C, D, and the outlet line 416B to the server manifold 408.

FIG. 4B is a block diagram illustrating rack-level 450A and server-level450B features of a cooling system incorporating a dielectric-basedcooling subsystem, along with the repurposable refrigerant coolingsubsystem in a multiple mode cooling subsystem, according to at leastone embodiment. In at least one embodiment, the dielectric-based coolingsubsystem is a dielectric refrigerant-based cooling subsystem and issupported by a dielectric refrigerant from the repurposable refrigerantcooling subsystem, which reduces the footprint of the multiple modecooling subsystem. The dielectric-based cooling subsystem enablesimmersion cooling of computing components in one or more servers 456 ofthe racks 452. The rack-level features 450A supporting or incorporatingthe dielectric-based cooling subsystem includes a rack 452 having inletand outlet dielectric-based manifolds 478A, 478B, having associatedtherewith inlet and outlet couplers or adapters 474, 476, and havingassociated therewith individual flow controllers 482 (two are referencedin the figure). In at least one embodiment, when the dielectric coolingmedia is a dielectric refrigerant, then the cooling componentsassociated with the refrigerant-based cooling subsystem may be used tosupport the dielectric-based cooling subsystem. The inlet and outletcouplers or adapters 474, 476 receive, direct, send dielectric coolingmedia 472 through one or more requesting (or indicating) servers 456;454.

In at least one embodiment, all the servers 456 in the rack 452 may beaddressed by their individual requests (or indications) for the samecooling requirement, or by a request (or indication) from the rack 452on behalf of its servers 456, which can be addressed by thedielectric-based cooling subsystem. Dielectric cooling media 480 exitsthe rack via adapter or coupler 476. The individual flow controllers 483may be pumps that are controlled (activated, deactivated, or maintainedin flow position) via inputs from the distributed or centralized controlsystem, in at least one embodiment. Further, in at least one embodiment,when the rack 452 makes its request (or providing its indication) for acooling requirement, all the flow controllers 482 may be simultaneouslycontrolled to address the cooling requirement, as if the coolingrequirement is representative of cooling requirements of all the servers456 at the same time.

In at least one embodiment, the request or the indication is made from asensor associated with the rack or with the servers. There may be one ormore sensors in different locations of the rack and in differentlocations of the server. The sensors, in a distributed configuration,may send individual requests or indications of a temperature, in atleast one embodiment. In at least one embodiment, the sensors work in acentralized configuration and are adapted to send its requests orindications to one sensor or to a centralized control system thataggregates the requests or indications to determine a singular action.In at least one embodiment, the singular action may be to send one ormore cooling media from the repurposable refrigerant cooling subsystemto address the requirements or indications associated with the singularaction. In at least one embodiment, this enables the present coolingsystem to send cooling media associated with a higher cooling capabilitythan the aggregate cooling requirement of some of the servers in theracks as the aggregated requested or indications may include at leastsome cooling requirements that are higher than the aggregate coolingrequirement.

In at least one embodiment, a mode statistical measure is used as anaggregate to indicate the number of servers having higher coolingrequirement, which may be addressed instead of a pure average of thecooling requirements of the servers. As such, it is appreciated that asubsystem offering higher cooling capability (such as immersion orrefrigerant-based cooling subsystem) may be activated when at least 50%of the servers of a rack have higher cooling requirements. In at leastone embodiment, a subsystem offering lower cooling capability (such as arefrigerant or a coolant-based cooling subsystem) may be activated whenat most 50% of the servers of the rack have higher cooling requirements.However, when a number of the servers of the rack start requesting orindicating higher cooling requirements after the refrigerant orcoolant-based cooling subsystem is activated for the rack, then a highercooling capability (such as from the dielectric-based cooling subsystem)may be provided to address the shortfall, if the servers (and computingcomponents) at issue can support immersion cooling. In at least oneembodiment, the change of the specific cooling subsystem may occur tohigher or lower capability cooling based in part on an amount of timeelapsed since a first request or indication occurred and when the firstrequest or indicate persists or shows an elevation of an associatedtemperature.

In at least one embodiment, the servers 456 have individual server-levelmanifolds, such as manifold 458 in server 454 (also referred to asserver tray or server box). As the server requires specific adaptationto support immersion cooling, such as being hermetically sealed, in atleast one embodiment. The manifold 458 of server 454 is adapted to floodthe server box for immersion cooling. The server manifold 338 receivesthe dielectric-based cooling media from rack manifold 478A (via adapteror coupler 468) and, depending on requirements of the computingcomponents associated with cold plates 462A-D, is caused to flood thedielectric-based cooling media into the server 454, so that thecomponents are directly in contact with the dielectric-based coolingmedia. As such, in at least one embodiment, the cold plates 462A-D maynot be required. In at least one embodiment, the distributed orcentralized control system associated with the repurposable refrigerantcooling subsystem is able to control the rack level flow controllers 482to facilitate movement of dielectric-based cooling media from thedielectric-based cooling subsystem. As such, in at least one embodiment,a sensor associated with the cold plates 462A-D may indicate or requesthigher cooling requirements than presently supported by, any of thecoolant, refrigerant, or evaporative cooling media. Then thedielectric-based cooling media is enabled to flow, via the appropriateflow controller 482 and via inlet adapter or coupler 468, to the server454, and out of outlet adapter or coupler 470. Each of the inlet and theoutlet adapter or coupler 468; 470 includes a one-way valve, so that thedielectric-based cooling media flows into the server box through oneadapter or coupler 468 and out of the other adapter or coupler 470.

In at least one embodiment, the server manifold 458 as well as the rowmanifold(s) 478A, B include multiple sub-manifolds or lines andassociated couplers to support the other cooling media, including thecoolant and the refrigerant (if the dielectric cooling media is not thesame as the refrigerant), but is illustrated and is discussed here forthe dielectric-based cooling media portion as relevant to this feature.In at least one embodiment, dielectric-based cooling media may enter therack manifold via adapter or coupler 474 and the heated or useddielectric-based cooling media 480 exits the rack via adapter or coupler476.

FIG. 5 is a process flow of steps available for a method 500 of using ormaking the cooling system of FIGS. 2A-4B and 6A-17D, according to atleast one embodiment. In at least one embodiment, the method for coolinga datacenter includes a step 502 for providing an evaporative coolingsubsystem to provide blown air for the datacenter according to a firstheat feature in the datacenter. Step 504 of the method 500 provides arepurposable refrigerant cooling subsystem that controls moisture of theblown air in a first configuration of the repurposable refrigerantcooling subsystem. Step 506 determines, using one or more sensorsdeployed in different servers or racks that a second heat feature isindicated. When, in step 508, it is determined that the second heatfeature requires more cooling capacity than the first heat feature, therepurposable refrigerant cooling subsystem is activated, in step 510, tocool the datacenter in a second configuration according to the secondheat feature in the datacenter. Step 506 can otherwise continue tomonitor the heat features of the servers or racks via the one or moresensors.

In at least one embodiment, the step 502 is further supported by a stepof guiding the blown air from the evaporative cooling subsystem to atleast one first rack of the datacenter using first looping componentsand according to a first heat feature or a first cooling requirement ofthe at least one first rack. In at least one embodiment, the step 510 isfurther supported by a step of guiding a refrigerant from therepurposable refrigerant cooling subsystem to the at least one firstrack of the datacenter using second looping components and according toa second heat feature or a second cooling requirement of the at leastone first rack.

In at least one embodiment, the step 510 includes a further step ofenabling the repurposable refrigerant cooling subsystem to function inthe second configuration when the evaporative cooling subsystem isineffective in achieving a reduction in heat generated in at least oneserver or rack of the datacenter. In at least one embodiment, the step510 enables the second configuration of the repurposable refrigerantcooling subsystem using outputs to flow controllers from a centralizedor distributed control system having at least one processor to providethe outputs.

In at least one embodiment, the step 510 includes a further step ofenabling one or more of the evaporative cooling subsystem, therepurposable refrigerant cooling subsystem, and a dielectric-basedcooling subsystem, each having different cooling capabilities for thedatacenter, using a refrigerant-based heat transfer subsystem. In atleast one embodiment, as illustrated in FIG. 2C, different pipes andfeatures are provided to enable refrigerant from the repurposablerefrigerant cooling subsystem to reach the evaporative cooling subsystemto control moisture in the blown air; to enable the refrigerant to reachthe racks, servers, and components via refrigerant manifolds; and toenable the refrigerant to reach the immersion cooling subsystem forflooding a supported server box.

In at least one embodiment, a flow controller may be provided on an exitside of the piping in any of the datacenter features. Furthermore, aflow controller may be provided on both the entry and the exit sides ofthe piping in any of the datacenter features. As such, when the flowcontroller is on the exit side, it performs a suction action rather thana pushing action. When two flow controllers are working in tandem, thereare both, suction and pushing actions. The flow rates achieved may behigher by the tandem flow controllers. The configuration or adaption ofthe flow controllers may be determined by requirements of thecomponents, for instance.

In at least one embodiment, the coolant-based secondary cooling loop ofthe embodiment in FIG. 3A facilitates a default or standard movement ofcoolant in a component-level cooling loop (such as enabled by piping344A, 346A in the server 334) or of a second coolant if the secondarycooling loop is directly available to a component. In at least oneembodiment, a learning subsystem may be implemented via the deeplearning application processor, such as processor 1400 in FIG. 14 andmay use the neurons 1502 and components thereof implemented usingcircuitry or logic, including one or more arithmetic logic units (ALUs)as described in FIG. 15. As such, the learning subsystem includes atleast one processor for evaluating temperatures within servers of theone or more racks with flow rates of different cooling media. Thelearning subsystem also provides outputs associated with at least twotemperatures for facilitating movement of two cooling media of thedifferent cooling media by controlling flow controllers associated withthe two cooling media to address the different cooling requirements.

Once trained, the learning subsystem will be able to provide one or moreoutputs having instructions associated with flow rates for the differentcooling media to the flow controllers. The outputs are provided after anevaluation of the prior associated flow rates of individual coolingmedia of the different cooling media and prior associated temperatures,as described in reference to FIGS. 2A-5. In at least one embodiment, thetemperatures used in the learning subsystem may be an inferred valuefrom a voltage and/or current value output by a respective component,such as a sensor, but the voltage and/or the current value itself may beused to train the learning subsystem, along with the requisite flow raterequired of the different cooling media to cause one or more computingcomponents, servers, racks, or associated sensors to achieve thetemperatures. Alternatively, the temperatures reflect one or moretemperatures at which the requisite flow rates for the cooling media isrequired of the cooling media, over what is already provided by thecooling media to cool the one or more computing components, servers,racks, or associated sensors, and subsequent temperatures may be used inthe learning subsystem to reflect the temperatures at which therequisite flow rates for the requisite cooling media are no longerrequired.

Alternatively, in at least one embodiment, the difference in thetemperatures and the subsequent temperatures of the different coolingmedia from the different cooling subsystems, along with the requisiteflow rates is used to train the learning system to recognize when toactivate and deactivate associated flow controllers for the differentcooling media. Once trained, the learning subsystem, through a devicecontroller (also referred to herein as a centralized control system ordistributed control system) described elsewhere throughout thisdisclosure, is able to provide one or more outputs to control theassociated flow controllers of the different cooling subsystems inresponse to temperature received from temperature sensors associatedwith different components, different servers, and different racks.

Furthermore, the learning subsystem executes a machine learning modelthat processes temperatures collected from prior applications, perhapsin a test environment, to control the temperatures within the differentservers and racks, and associated with different components. Thecollected temperatures may include (a) achieved temperatures for certainflow rates of the different cooling media; (b) achieved difference intemperatures for certain flow rates of the different cooling media forspecific time periods; and (c) initial temperatures and correspondingflow rates of the different media applied to keep the different servers,racks, or components in optimal operation by their different coolingrequirements. In at least one embodiment, the collected information mayinclude a reference to a coolant or a refrigerant type, or acompatibility type (such as server or rack having immersion-coolingcompatibility), which reflects the type of cooling required oravailable.

In at least one embodiment, aspects of the processing for the deeplearning subsystem may use the collected information processed in linewith the features discussed with reference to FIGS. 14, 15. In anexample, the processing of the temperatures uses multiple neuron levelsof the machine learning model that are loaded with one or more of thecollected temperature features noted above and the corresponding flowrates of the different cooling media. The learning subsystem performs atraining that may be represented as an evaluation of temperature changesassociated with prior flow rates (or changes in flow rates) of eachcooling media as per adjustments made to the one or more flowcontrollers associated with the different cooling media. The neuronlevels may store values associated with the evaluation process and mayrepresent an association or correlation between the temperature changesand the flow rates for each of the different cooling media.

In at least one embodiment, the learning subsystem, once trained, isable to determine, in application, flow rates of the different mediarequired to achieve cooling to a temperature (or a change, such as areduction in temperature) associated with cooling requirements ofdifferent servers, different racks, or different components, forinstance. As the cooling requirements have to be within the coolingcapabilities of the appropriate cooling subsystem, the learningsubsystem is able to select which of the cooling subsystems to activateto address the temperature sensed from the different servers, thedifferent racks, or the different components. The collected temperaturesand prior associated flow rates for the different cooling media that isused to achieve the collected temperatures (or differences, forinstance) may be used by the learning subsystem to provide one or moreoutputs associated with required flow rates of different cooling mediato address the different cooling requirements reflected by a temperature(representing a reduction) than a present temperature for the differentservers, the different racks, or the different components.

In at least one embodiment, a result of the learning subsystem is, inresponse to a sensed temperature from the different servers, thedifferent racks, or the different components, one or more outputs to theflow controllers associated with the different cooling subsystems thatmodifies flow rates of the respective different cooling media. Themodification of the flow rates enables a determined flow of therespective different cooling media to reach the different servers, thedifferent racks, or the different components requiring the cooling. Themodified flow rate may be maintained till the temperature in therequiring area reaches a temperature associated with the flow rate ofthe cooling media that is known to the learning subsystem. In at leastone embodiment, the modified flow rate may be maintained till thetemperature in the area changes by a determined value. In at least oneembodiment, the modified flow rate may be maintained till thetemperature in the area reaches a rated temperature for the differentservers, different racks, the different components, or the differentcooling media.

In at least one embodiment, the device controller (also referred to as acentralized or distributed control system) includes at least oneprocessor having at least one logic unit to control flow controllersassociated with the different cooling subsystems. In at least oneembodiment, the device controller may be at least one processor withinthe datacenter, such as a processor 702 of FIG. 7A. The flow controllerfacilitates movement of a respective cooling media associated with arespective cooling subsystem, and facilitates cooling of areas in thedatacenter in response to temperatures sensed in the area. In at leastone embodiment, the at least one processor is a processor core of amulti-core processor, such as multi-core processors 905, 906 in FIG. 9A.In at least one embodiment, the at least one logic unit may be adaptedto receive a temperature values from temperature sensors associated withservers or the one or more racks, and is adapted to facilitate movementof a first individual cooling media and a second individual coolingmedia of the at least two cooling media.

In at least one embodiment, a processor, such as the processor cores ofmulti-core processors 905, 906 in FIG. 9A may include a learningsubsystem for evaluating temperatures from sensors in differentlocations in the datacenter, such as in different locations associatedwith the servers or the racks, or even a component within the server,with flow rates that are associated with at least two cooling media of arepurposable refrigerant cooling subsystem having two or more of anevaporative cooling subsystem, a liquid-based cooling subsystem, adielectric-based cooling subsystem, or a repurposable refrigerantcooling subsystem. The learning subsystem provides an output, such as aninstruction that is associated with at least two temperatures forfacilitating movement of at least two cooling media by controlling theflow controllers associated with the at least two cooling media toaddress the different cooling requirements.

In at least one embodiment, the learning subsystem executes a machinelearning model to process the temperature using multiple neuron levelsof the machine learning model having the temperatures and having priorassociated flow rates for the different cooling media. The machinelearning model may be a implemented using the neurons structuredescribed in FIG. 15 and the deep learning processor as described inFIG. 14. The machine learning model provides the output associated withthe flow rate, from an evaluation of the prior associated flow rates, tothe one or more flow controllers. Furthermore, an instruction output ofthe processor, such as a pin of a connector bus or a ball of a ball gridarray, enables communication of the output with the one or more flowcontrollers to modify a first flow rate of a first cooling media of theat least two different cooling media, while maintaining a second flowrate of a second individual cooling media.

In at least one embodiment, the present disclosure is to at least oneprocessor for a cooling system or to a system having the at least oneprocessor. The at least one processor includes at least one logic unitto train a neural network having hidden layers of neurons for evaluatingtemperatures and flow rates that are associated with at least twocooling media of a repurposable refrigerant cooling subsystem comprisingtwo or more of an evaporative cooling subsystem, a liquid-based coolingsubsystem, a dielectric-based cooling subsystem, or a repurposablerefrigerant cooling subsystem. The repurposable refrigerant coolingsubsystem is rated for different cooling capacities and is adjustable todifferent cooling requirements of the datacenter within a minimum and amaximum of the different cooling capacities. The cooling system has aform factor for one or more racks of a datacenter.

As described elsewhere in this disclosure and with references made toFIGS. 2A-5, 14, and 15, the training may be performed by layers ofneurons that are provided inputs of the temperatures of one or moreareas in the datacenter and associated flow rates of different coolingmedia from prior application, perhaps in a test environment. Thetemperatures may include a starting temperature (and associated flowrate of each cooling media applied for the starting temperature to bringthe temperature to a rated temperature of the one or more areas), afinal temperature (after each cooling media is applied, over eachcooling media already in the area, for a flow rate for a period oftime), and a difference in temperatures achieved for the flow rate andthe period of time of application of the different cooling media. One ormore of these features may be used to train a neural network todetermine when to apply a flow of the different cooling media and/orwhen to stop the flow of the different cooling media for cooling—such aswhen a temperature sensed for an area reaches the starting temperature,when the temperature sensed for the area reaches the final temperature,and/or when the temperature sensed for the area reflects the differencein temperature suited for the area (such as a reduction in temperatureto a rated temperature).

In at least one embodiment, the at least one processor includes at leastone logic unit configured or adapted for training a neural network andis a multi-core processor. In at least one embodiment, the at least onelogic unit may be within one processor core that can be used to forevaluate temperatures and flow rates that are associated with at leasttwo cooling media of a repurposable refrigerant cooling subsystem havingtwo or more of an evaporative cooling subsystem, a liquid-based coolingsubsystem, a dielectric-based cooling subsystem, or a repurposablerefrigerant cooling subsystem. The repurposable refrigerant coolingsubsystem is rated for different cooling capacities and is adjustable todifferent cooling requirements of the datacenter within a minimum and amaximum of the different cooling capacities. The cooling system has aform factor for one or more racks of a datacenter.

In at least one embodiment, the present disclosure enables all-in-oneair, liquid, refrigeration, and immersion cooling system that is in aform factor of an individual server rack or multiple racks and thatprovides required cooling with an evaporative cooling subsystem having aheat exchanger associated with a coolant, with a warm or cold waterliquid-based cooling subsystem for chip cooling, with a refrigeration toheat exchanger repurposable refrigerant cooling subsystem, and with animmersion blade cooling reflecting a dielectric-based cooling subsystem.The repurposable refrigerant cooling subsystem in a rack is part of acooling system that can provide cooling by passing air through cooledheat exchangers, but can also provide cooling water from a primaryliquid supply, and can also provide a vapor compression cooling systemfor refrigeration cooling with an associated evaporator heat exchangerfor either server or chip cooling. In addition, a coolant or refrigerantthat is of dielectric properties can be used either from theliquid-based cooling subsystem or the repurposable refrigerant (orphase)-based cooling subsystem for removing thermal load from animmersion blade cooling server in a rack.

In at least one embodiment, the present disclosure is therefore anall-in-one rack cooling system that can provide cooling from arelatively low density rack having about 10 KW to higher density coolingof about 30 KW using the air-base cooling subsystem; from about 30 KW to60 KW using a liquid-based cooling subsystem; from about 60 KW to 100 kWusing the repurposable refrigerant cooling subsystem, and from about 100KW to 450 KW using the dielectric-based cooling subsystem for immersioncooling. The all-in-one rack is an all in one structure and reflects afoot print that can be readily deployed in almost all data centers.

Datacenter

FIG. 6A illustrates an example datacenter 600, in which at least oneembodiment from FIGS. 2A-5 may be used. In at least one embodiment,datacenter 600 includes a datacenter infrastructure layer 610, aframework layer 620, a software layer 630, and an application layer 640.In at least one embodiment, such as described in respect to FIGS. 2A-5,features in the components of the cooling subsystems of the repurposablerefrigerant cooling subsystem may be performed inside or incollaboration with the example datacenter 600. In at least oneembodiment, the infrastructure layer 610, the framework layer 620, thesoftware layer 630, and the application layer 640 may be partly or fullyprovided via computing components on server trays located in racks 210of the datacenter 200. This enables cooling systems of the presentdisclosure to direct cooling to certain ones of the computing componentsin an efficient and effective manner. Further, aspects of thedatacenter, including the datacenter infrastructure layer 610, theframework layer 620, the software layer 630, and the application layer640 may be used to support the refrigerant-assisted cooling discussedwith at least reference to FIGS. 2A-5 above. As such, the discussion inreference to FIGS. 6A-17D may be understood to apply to the hardware andsoftware features required to enable or support the repurposablerefrigerant cooling subsystem for the datacenter of FIGS. 2A-5, forinstance.

In at least one embodiment, as in FIG. 6A, datacenter infrastructurelayer 610 may include a resource orchestrator 612, grouped computingresources 614, and node computing resources (“node C.R.s”)616(1)-616(N), where “N” represents any whole, positive integer. In atleast one embodiment, node C.R.s 616(1)-616(N) may include, but are notlimited to, any number of central processing units (“CPUs”) or otherprocessors (including accelerators, field programmable gate arrays(FPGAs), graphics processors, etc.), memory devices (such as dynamicread-only memory), storage devices (such as solid state or disk drives),network input/output (“NW I/O”) devices, network switches, virtualmachines (“VMs”), power modules, and cooling modules, etc. In at leastone embodiment, one or more node C.R.s from among node C.R.s616(1)-616(N) may be a server having one or more of above-mentionedcomputing resources.

In at least one embodiment, grouped computing resources 614 may includeseparate groupings of node C.R.s housed within one or more racks (notshown), or many racks housed in datacenters at various geographicallocations (also not shown). Separate groupings of node C.R.s withingrouped computing resources 614 may include grouped compute, network,memory or storage resources that may be configured or allocated tosupport one or more workloads. In at least one embodiment, several nodeC.R.s including CPUs or processors may grouped within one or more racksto provide compute resources to support one or more workloads. In atleast one embodiment, one or more racks may also include any number ofpower modules, cooling modules, and network switches, in anycombination.

In at least one embodiment, resource orchestrator 612 may configure orotherwise control one or more node C.R.s 616(1)-616(N) and/or groupedcomputing resources 614. In at least one embodiment, resourceorchestrator 612 may include a software design infrastructure (“SDI”)management entity for datacenter 600. In at least one embodiment,resource orchestrator may include hardware, software or some combinationthereof.

In at least one embodiment, as shown in FIG. 6A, framework layer 620includes a job scheduler 622, a configuration manager 624, a resourcemanager 626 and a distributed file system 628. In at least oneembodiment, framework layer 620 may include a framework to supportsoftware 632 of software layer 630 and/or one or more application(s) 642of application layer 640. In at least one embodiment, software 632 orapplication(s) 642 may respectively include web-based service softwareor applications, such as those provided by Amazon Web Services, GoogleCloud and Microsoft Azure. In at least one embodiment, framework layer620 may be, but is not limited to, a type of free and open-sourcesoftware web application framework such as Apache Spark™ (hereinafter“Spark”) that may utilize distributed file system 628 for large-scaledata processing (such as “big data”). In at least one embodiment, jobscheduler 622 may include a Spark driver to facilitate scheduling ofworkloads supported by various layers of datacenter 600. In at least oneembodiment, configuration manager 624 may be capable of configuringdifferent layers such as software layer 630 and framework layer 620including Spark and distributed file system 628 for supportinglarge-scale data processing. In at least one embodiment, resourcemanager 626 may be capable of managing clustered or grouped computingresources mapped to or allocated for support of distributed file system628 and job scheduler 622. In at least one embodiment, clustered orgrouped computing resources may include grouped computing resource 614at datacenter infrastructure layer 610. In at least one embodiment,resource manager 626 may coordinate with resource orchestrator 612 tomanage these mapped or allocated computing resources.

In at least one embodiment, software 632 included in software layer 630may include software used by at least portions of node C.R.s616(1)-616(N), grouped computing resources 614, and/or distributed filesystem 628 of framework layer 620. One or more types of software mayinclude, but are not limited to, Internet web page search software,e-mail virus scan software, database software, and streaming videocontent software.

In at least one embodiment, application(s) 642 included in applicationlayer 640 may include one or more types of applications used by at leastportions of node C.R.s 616(1)-616(N), grouped computing resources 614,and/or distributed file system 628 of framework layer 620. One or moretypes of applications may include, but are not limited to, any number ofa genomics application, a cognitive compute, and a machine learningapplication, including training or inferencing software, machinelearning framework software (such as PyTorch, TensorFlow, Caffe, etc.)or other machine learning applications used in conjunction with one ormore embodiments.

In at least one embodiment, any of configuration manager 624, resourcemanager 626, and resource orchestrator 612 may implement any number andtype of self-modifying actions based on any amount and type of dataacquired in any technically feasible fashion. In at least oneembodiment, self-modifying actions may relieve a datacenter operator ofdatacenter 600 from making possibly bad configuration decisions andpossibly avoiding underutilized and/or poor performing portions of adatacenter.

In at least one embodiment, datacenter 600 may include tools, services,software or other resources to train one or more machine learning modelsor predict or infer information using one or more machine learningmodels according to one or more embodiments described herein. In atleast one embodiment, in at least one embodiment, a machine learningmodel may be trained by calculating weight parameters according to aneural network architecture using software and computing resourcesdescribed above with respect to datacenter 600. In at least oneembodiment, trained machine learning models corresponding to one or moreneural networks may be used to infer or predict information usingresources described above with respect to datacenter 600 by using weightparameters calculated through one or more training techniques describedherein. As previously discussed, deep learning techniques may be used tosupport intelligent control of the flow controllers in therefrigerant-assisted cooling by monitoring area temperatures of thedatacenter. Deep learning may be advanced using any appropriate learningnetwork and the computing capabilities of the datacenter 600. As such, adeep neural network (DNN), a recurrent neural network (RNN) or aconvolutional neural network (CNN) may be supported eithersimultaneously or concurrently using the hardware in the datacenter.Once a network is trained and successfully evaluated to recognize datawithin a subset or a slice, for instance, the trained network canprovide similar representative data for using with the collected data.

In at least one embodiment, datacenter 600 may use CPUs,application-specific integrated circuits (ASICs), GPUs, FPGAs, or otherhardware to perform training and/or inferencing using above-describedresources. Moreover, one or more software and/or hardware resourcesdescribed above may be configured as a service to allow users to trainor performing inferencing of information, such as pressure, flow rates,temperature, and location information, or other artificial intelligenceservices.

Inference and Training Logic

Inference and/or training logic 615 may be used to perform inferencingand/or training operations associated with one or more embodiments. Inat least one embodiment, inference and/or training logic 615 may be usedin system FIG. 6A for inferencing or predicting operations based, atleast in part, on weight parameters calculated using neural networktraining operations, neural network functions and/or architectures, orneural network use cases described herein. In at least one embodiment,inference and/or training logic 615 may include, without limitation,hardware logic in which computational resources are dedicated orotherwise exclusively used in conjunction with weight values or otherinformation corresponding to one or more layers of neurons within aneural network. In at least one embodiment, inference and/or traininglogic 615 may be used in conjunction with an application-specificintegrated circuit (ASIC), such as Tensorflow® Processing Unit fromGoogle, an inference processing unit (IPU) from Graphcore™, or aNervana® (such as “Lake Crest”) processor from Intel Corp.

In at least one embodiment, inference and/or training logic 615 may beused in conjunction with central processing unit (CPU) hardware,graphics processing unit (GPU) hardware or other hardware, such as fieldprogrammable gate arrays (FPGAs). In at least one embodiment, inferenceand/or training logic 615 includes, without limitation, code and/or datastorage modules which may be used to store code (such as graph code),weight values and/or other information, including bias values, gradientinformation, momentum values, and/or other parameter or hyperparameterinformation. In at least one embodiment, each of the code and/or datastorage modules is associated with a dedicated computational resource.In at least one embodiment, the dedicated computational resourceincludes computational hardware that further include one or more ALUsthat perform mathematical functions, such as linear algebraic functions,only on information stored in code and/or data storage modules, andresults from which are stored in an activation storage module of theinference and/or training logic 615.

FIGS. 6B, 6C illustrates inference and/or training logic, such as usedin FIG. 6A and in at least one embodiment of the present disclosure,according to at least one embodiment. The inference and/or traininglogic 615 are used to perform inferencing and/or training operationsassociated with at least one embodiment. Details regarding inferenceand/or training logic 615 are provided below in conjunction with FIGS.6B and/or 6C. The inference and/or training logic 615 of FIGS. 6B and 6Care distinguished by the use of the arithmetic logic units (ALUs) 610versus the computational hardware 602, 606. In at least one embodiment,each of computational hardware 602 and computational hardware 606includes one or more ALUs that perform mathematical functions, such aslinear algebraic functions, only on information stored in code and/ordata storage 601 and code and/or data storage 605, respectively, resultof which is stored in activation storage 620. As such, FIGS. 6B and 6Cmay be alternatives and may be used interchangeably unless statedotherwise.

In at least one embodiment, inference and/or training logic 615 mayinclude, without limitation, code and/or data storage 601 to storeforward and/or output weight and/or input/output data, and/or otherparameters to configure neurons or layers of a neural network trainedand/or used for inferencing in at least one embodiment. In at least oneembodiment, training logic 615 may include, or be coupled to code and/ordata storage 601 to store graph code or other software to control timingand/or order, in which weight and/or other parameter information is tobe loaded to configure, logic, including integer and/or floating pointunits (collectively, arithmetic logic units (ALUs). In at least oneembodiment, code, such as graph code, loads weight or other parameterinformation into processor ALUs based on an architecture of a neuralnetwork to which the code corresponds. In at least one embodiment, codeand/or data storage 601 stores weight parameters and/or input/outputdata of each layer of a neural network trained or used in conjunctionwith at least one embodiment during forward propagation of input/outputdata and/or weight parameters during training and/or inferencing usingat least one embodiment. In at least one embodiment, any portion of codeand/or data storage 601 may be included with other on-chip or off-chipdata storage, including a processor's L1, L2, or L3 cache or systemmemory.

In at least one embodiment, any portion of code and/or data storage 601may be internal or external to one or more processors or other hardwarelogic devices or circuits. In at least one embodiment, code and/or codeand/or data storage 601 may be cache memory, dynamic randomlyaddressable memory (“DRAM”), static randomly addressable memory(“SRAM”), non-volatile memory (such as Flash memory), or other storage.In at least one embodiment, choice of whether code and/or code and/ordata storage 601 is internal or external to a processor, for example, orincluded of DRAM, SRAM, Flash or some other storage type may depend onavailable storage on-chip versus off-chip, latency requirements oftraining and/or inferencing functions being performed, batch size ofdata used in inferencing and/or training of a neural network, or somecombination of these factors.

In at least one embodiment, inference and/or training logic 615 mayinclude, without limitation, a code and/or data storage 605 to storebackward and/or output weight and/or input/output data corresponding toneurons or layers of a neural network trained and/or used forinferencing in at least one embodiment. In at least one embodiment, codeand/or data storage 605 stores weight parameters and/or input/outputdata of each layer of a neural network trained or used in conjunctionwith at least one embodiment during backward propagation of input/outputdata and/or weight parameters during training and/or inferencing usingat least one embodiment. In at least one embodiment, training logic 615may include, or be coupled to code and/or data storage 605 to storegraph code or other software to control timing and/or order, in whichweight and/or other parameter information is to be loaded to configure,logic, including integer and/or floating point units (collectively,arithmetic logic units (ALUs).

In at least one embodiment, code, such as graph code, loads weight orother parameter information into processor ALUs based on an architectureof a neural network to which the code corresponds. In at least oneembodiment, any portion of code and/or data storage 605 may be includedwith other on-chip or off-chip data storage, including a processor's L1,L2, or L3 cache or system memory. In at least one embodiment, anyportion of code and/or data storage 605 may be internal or external toon one or more processors or other hardware logic devices or circuits.In at least one embodiment, code and/or data storage 605 may be cachememory, DRAM, SRAM, non-volatile memory (such as Flash memory), or otherstorage. In at least one embodiment, choice of whether code and/or datastorage 605 is internal or external to a processor, for example, orincluded of DRAM, SRAM, Flash or some other storage type may depend onavailable storage on-chip versus off-chip, latency requirements oftraining and/or inferencing functions being performed, batch size ofdata used in inferencing and/or training of a neural network, or somecombination of these factors.

In at least one embodiment, code and/or data storage 601 and code and/ordata storage 605 may be separate storage structures. In at least oneembodiment, code and/or data storage 601 and code and/or data storage605 may be same storage structure. In at least one embodiment, codeand/or data storage 601 and code and/or data storage 605 may bepartially same storage structure and partially separate storagestructures. In at least one embodiment, any portion of code and/or datastorage 601 and code and/or data storage 605 may be included with otheron-chip or off-chip data storage, including a processor's L1, L2, or L3cache or system memory.

In at least one embodiment, inference and/or training logic 615 mayinclude, without limitation, one or more arithmetic logic unit(s)(“ALU(s)”) 610, including integer and/or floating point units, toperform logical and/or mathematical operations based, at least in parton, or indicated by, training and/or inference code (such as graphcode), a result of which may produce activations (such as output valuesfrom layers or neurons within a neural network) stored in an activationstorage 620 that are functions of input/output and/or weight parameterdata stored in code and/or data storage 601 and/or code and/or datastorage 605. In at least one embodiment, activations stored inactivation storage 620 are generated according to linear algebraic andor matrix-based mathematics performed by ALU(s) 610 in response toperforming instructions or other code, wherein weight values stored incode and/or data storage 605 and/or code and/or data storage 601 areused as operands along with other values, such as bias values, gradientinformation, momentum values, or other parameters or hyperparameters,any or all of which may be stored in code and/or data storage 605 orcode and/or data storage 601 or another storage on or off-chip.

In at least one embodiment, ALU(s) 610 are included within one or moreprocessors or other hardware logic devices or circuits, whereas inanother embodiment, ALU(s) 610 may be external to a processor or otherhardware logic device or circuit that uses them (such as aco-processor). In at least one embodiment, ALUs 610 may be includedwithin a processor's execution units or otherwise within a bank of ALUsaccessible by a processor's execution units either within same processoror distributed between different processors of different types (such ascentral processing units, graphics processing units, fixed functionunits, etc.). In at least one embodiment, code and/or data storage 601,code and/or data storage 605, and activation storage 620 may be on sameprocessor or other hardware logic device or circuit, whereas in anotherembodiment, they may be in different processors or other hardware logicdevices or circuits, or some combination of same and differentprocessors or other hardware logic devices or circuits. In at least oneembodiment, any portion of activation storage 620 may be included withother on-chip or off-chip data storage, including a processor's L1, L2,or L3 cache or system memory. Furthermore, inferencing and/or trainingcode may be stored with other code accessible to a processor or otherhardware logic or circuit and fetched and/or processed using aprocessor's fetch, decode, scheduling, execution, retirement and/orother logical circuits.

In at least one embodiment, activation storage 620 may be cache memory,DRAM, SRAM, non-volatile memory (such as Flash memory), or otherstorage. In at least one embodiment, activation storage 620 may becompletely or partially within or external to one or more processors orother logical circuits. In at least one embodiment, choice of whetheractivation storage 620 is internal or external to a processor, forexample, or included of DRAM, SRAM, Flash or some other storage type maydepend on available storage on-chip versus off-chip, latencyrequirements of training and/or inferencing functions being performed,batch size of data used in inferencing and/or training of a neuralnetwork, or some combination of these factors. In at least oneembodiment, inference and/or training logic 615 illustrated in FIG. 6Bmay be used in conjunction with an application-specific integratedcircuit (“ASIC”), such as Tensorflow® Processing Unit from Google, aninference processing unit (IPU) from Graphcore™, or a Nervana® (such as“Lake Crest”) processor from Intel Corp. In at least one embodiment,inference and/or training logic 615 illustrated in FIG. 6B may be usedin conjunction with central processing unit (“CPU”) hardware, graphicsprocessing unit (“GPU”) hardware or other hardware, such as fieldprogrammable gate arrays (“FPGAs”).

In at least one embodiment, as illustrated in FIG. 6C, inference and/ortraining logic 615 may include, without limitation, hardware logic inwhich computational resources are dedicated or otherwise exclusivelyused in conjunction with weight values or other informationcorresponding to one or more layers of neurons within a neural network.In at least one embodiment, inference and/or training logic 615illustrated in FIG. 6C may be used in conjunction with anapplication-specific integrated circuit (ASIC), such as Tensorflow®Processing Unit from Google, an inference processing unit (IPU) fromGraphcore™, or a Nervana® (such as “Lake Crest”) processor from IntelCorp. In at least one embodiment, inference and/or training logic 615illustrated in FIG. 6C may be used in conjunction with centralprocessing unit (CPU) hardware, graphics processing unit (GPU) hardwareor other hardware, such as field programmable gate arrays (FPGAs). In atleast one embodiment, inference and/or training logic 615 includes,without limitation, code and/or data storage 601 and code and/or datastorage 605, which may be used to store code (such as graph code),weight values and/or other information, including bias values, gradientinformation, momentum values, and/or other parameter or hyperparameterinformation. In at least one embodiment illustrated in FIG. 6C, each ofcode and/or data storage 601 and code and/or data storage 605 isassociated with a dedicated computational resource, such ascomputational hardware 602 and computational hardware 606, respectively.

In at least one embodiment, each of code and/or data storage 601 and 605and corresponding computational hardware 602 and 606, respectively,correspond to different layers of a neural network, such that resultingactivation from one “storage/computational pair 601/602” of code and/ordata storage 601 and computational hardware 602 is provided as an inputto “storage/computational pair 605/606” of code and/or data storage 605and computational hardware 606, in order to mirror conceptualorganization of a neural network. In at least one embodiment, each ofstorage/computational pairs 601/602 and 605/606 may correspond to morethan one neural network layer. In at least one embodiment, additionalstorage/computation pairs (not shown) subsequent to or in parallel withstorage computation pairs 601/602 and 605/606 may be included ininference and/or training logic 615.

Computer Systems

FIG. 7A is a block diagram illustrating an exemplary computer system700A, which may be a system with interconnected devices and components,a system-on-a-chip (SOC) or some combination thereof formed with aprocessor that may include execution units to execute an instruction tosupport and/or to enable the intelligent control of a multiple modecooling subsystem having a repurposable refrigerant cooling subsystem asdescribed herein, according to at least one embodiment. In at least oneembodiment, computer system 700A may include, without limitation, acomponent, such as a processor 702 to employ execution units includinglogic to perform algorithms for process data, in accordance with presentdisclosure, such as in embodiment described herein. In at least oneembodiment, computer system 700A may include processors, such asPENTIUM® Processor family, Xeon™, Itanium®, XScale™ and/or StrongARM™,Intel® Core™, or Intel® Nervana™ microprocessors available from IntelCorporation of Santa Clara, Calif., although other systems (includingPCs having other microprocessors, engineering workstations, set-topboxes and like) may also be used. In at least one embodiment, computersystem 700B may execute a version of WINDOWS' operating system availablefrom Microsoft Corporation of Redmond, Wash., although other operatingsystems (UNIX and Linux for example), embedded software, and/orgraphical user interfaces, may also be used.

In at least one embodiment, the exemplary computer system 700A mayincorporate one or more of components 110-116 (from FIG. 1) to supportprocessing aspects for the intelligent control for the repurposablerefrigerant cooling subsystem. For at least this reason, in oneembodiment, FIG. 7A illustrates a system, which includes interconnectedhardware devices or “chips”, whereas in other embodiments, FIG. 7A mayillustrate an exemplary System on a Chip (“SoC”). In at least oneembodiment, devices may be interconnected with proprietaryinterconnects, standardized interconnects (such as PCIe) or somecombination thereof. In at least one embodiment, one or more componentsof computer system 700B are interconnected using compute express link(CXL) interconnects. Inference and/or training logic 615 are used toperform inferencing and/or training operations associated with one ormore embodiments, as previously discussed with respect to FIGS. 6A-C,for instance. Details regarding inference and/or training logic 615 areprovided below in conjunction with FIGS. 6A-C. In at least oneembodiment, inference and/or training logic 615 may be used in systemFIG. 7A for inferencing or predicting operations based, at least inpart, on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

Embodiments may be used in other devices such as handheld devices andembedded applications. Some examples of handheld devices includecellular phones, Internet Protocol devices, digital cameras, personaldigital assistants (“PDAs”), and handheld PCs. In at least oneembodiment, embedded applications may include a microcontroller, adigital signal processor (“DSP”), system on a chip, network computers(“NetPCs”), set-top boxes, network hubs, wide area network (“WAN”)switches, or any other system that may perform one or more instructionsin accordance with at least one embodiment.

In at least one embodiment, computer system 700A may include, withoutlimitation, processor 702 that may include, without limitation, one ormore execution units 708 to perform machine learning model trainingand/or inferencing according to techniques described herein. In at leastone embodiment, computer system 700A is a single processor desktop orserver system, but in another embodiment computer system 700A may be amultiprocessor system. In at least one embodiment, processor 702 mayinclude, without limitation, a complex instruction set computer (“CISC”)microprocessor, a reduced instruction set computing (“RISC”)microprocessor, a very long instruction word (“VLIW”) microprocessor, aprocessor implementing a combination of instruction sets, or any otherprocessor device, such as a digital signal processor, for example. In atleast one embodiment, processor 702 may be coupled to a processor bus710 that may transmit data signals between processor 702 and othercomponents in computer system 700A.

In at least one embodiment, processor 702 may include, withoutlimitation, a Level 1 (“L1”) internal cache memory (“cache”) 704. In atleast one embodiment, processor 702 may have a single internal cache ormultiple levels of internal cache. In at least one embodiment, cachememory may reside external to processor 702. Other embodiments may alsoinclude a combination of both internal and external caches depending onparticular implementation and needs. In at least one embodiment,register file 706 may store different types of data in various registersincluding, without limitation, integer registers, floating pointregisters, status registers, and instruction pointer register.

In at least one embodiment, execution unit 708, including, withoutlimitation, logic to perform integer and floating point operations, alsoresides in processor 702. In at least one embodiment, processor 702 mayalso include a microcode (“ucode”) read only memory (“ROM”) that storesmicrocode for certain macro instructions. In at least one embodiment,execution unit 708 may include logic to handle a packed instruction set709. In at least one embodiment, by including packed instruction set 709in an instruction set of a general-purpose processor 702, along withassociated circuitry to execute instructions, operations used by manymultimedia applications may be performed using packed data in ageneral-purpose processor 702. In one or more embodiments, manymultimedia applications may be accelerated and executed more efficientlyby using full width of a processor's data bus for performing operationson packed data, which may eliminate need to transfer smaller units ofdata across processor's data bus to perform one or more operations onedata element at a time.

In at least one embodiment, execution unit 708 may also be used inmicrocontrollers, embedded processors, graphics devices, DSPs, and othertypes of logic circuits. In at least one embodiment, computer system700A may include, without limitation, a memory 720. In at least oneembodiment, memory 720 may be implemented as a Dynamic Random AccessMemory (“DRAM”) device, a Static Random Access Memory (“SRAM”) device,flash memory device, or other memory device. In at least one embodiment,memory 720 may store instruction(s) 719 and/or data 721 represented bydata signals that may be executed by processor 702.

In at least one embodiment, system logic chip may be coupled toprocessor bus 710 and memory 720. In at least one embodiment, systemlogic chip may include, without limitation, a memory controller hub(“MCH”) 716, and processor 702 may communicate with MCH 716 viaprocessor bus 710. In at least one embodiment, MCH 716 may provide ahigh bandwidth memory path 718 to memory 720 for instruction and datastorage and for storage of graphics commands, data and textures. In atleast one embodiment, MCH 716 may direct data signals between processor702, memory 720, and other components in computer system 700A and tobridge data signals between processor bus 710, memory 720, and a systemI/O 722. In at least one embodiment, system logic chip may provide agraphics port for coupling to a graphics controller. In at least oneembodiment, MCH 716 may be coupled to memory 720 through a highbandwidth memory path 718 and graphics/video card 712 may be coupled toMCH 716 through an Accelerated Graphics Port (“AGP”) interconnect 714.

In at least one embodiment, computer system 700A may use system I/O 722that is a proprietary hub interface bus to couple MCH 716 to I/Ocontroller hub (“ICH”) 730. In at least one embodiment, ICH 730 mayprovide direct connections to some I/O devices via a local I/O bus. Inat least one embodiment, local I/O bus may include, without limitation,a high-speed I/O bus for connecting peripherals to memory 720, chipset,and processor 702. Examples may include, without limitation, an audiocontroller 729, a firmware hub (“flash BIOS”) 728, a wirelesstransceiver 726, a data storage 724, a legacy I/O controller 723containing user input and keyboard interfaces 725, a serial expansionport 727, such as Universal Serial Bus (“USB”), and a network controller734. Data storage 724 may include a hard disk drive, a floppy diskdrive, a CD-ROM device, a flash memory device, or other mass storagedevice.

FIG. 7B is a block diagram illustrating an electronic device 700B forutilizing a processor 710 to support and/or to enable intelligentcontrol of a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem as described herein, according to at leastone embodiment. In at least one embodiment, electronic device 700B maybe, for example and without limitation, a notebook, a tower server, arack server, a blade server, a laptop, a desktop, a tablet, a mobiledevice, a phone, an embedded computer, or any other suitable electronicdevice. In at least one embodiment, the exemplary electronic device 700Bmay incorporate one or more of components that support processingaspects for the repurposable refrigerant cooling subsystem.

In at least one embodiment, system 700B may include, without limitation,processor 710 communicatively coupled to any suitable number or kind ofcomponents, peripherals, modules, or devices. In at least oneembodiment, processor 710 coupled using a bus or interface, such as a 1°C. bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, aSerial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”)bus, a Serial Advance Technology Attachment (“SATA”) bus, a UniversalSerial Bus (“USB”) (versions 1, 2, 3), or a Universal AsynchronousReceiver/Transmitter (“UART”) bus. In at least one embodiment, FIG. 7Billustrates a system, which includes interconnected hardware devices or“chips”, whereas in other embodiments, FIG. 7B may illustrate anexemplary System on a Chip (“SoC”). In at least one embodiment, devicesillustrated in FIG. 7B may be interconnected with proprietaryinterconnects, standardized interconnects (such as PCIe) or somecombination thereof. In at least one embodiment, one or more componentsof FIG. 7B are interconnected using compute express link (CXL)interconnects.

In at least one embodiment, FIG. 7B may include a display 724, a touchscreen 725, a touch pad 730, a Near Field Communications unit (“NFC”)745, a sensor hub 740, a thermal sensor 746, an Express Chipset (“EC”)735, a Trusted Platform Module (“TPM”) 738, BIOS/firmware/flash memory(“BIOS, FW Flash”) 722, a DSP 760, a drive 720 such as a Solid StateDisk (“SSD”) or a Hard Disk Drive (“HDD”), a wireless local area networkunit (“WLAN”) 750, a Bluetooth unit 752, a Wireless Wide Area Networkunit (“WWAN”) 756, a Global Positioning System (GPS) 755, a camera (“USB3.0 camera”) 754 such as a USB 3.0 camera, and/or a Low Power DoubleData Rate (“LPDDR”) memory unit (“LPDDR3”) 715 implemented in, forexample, LPDDR3 standard. These components may each be implemented inany suitable manner.

In at least one embodiment, other components may be communicativelycoupled to processor 710 through components discussed above. In at leastone embodiment, an accelerometer 741, Ambient Light Sensor (“ALS”) 742,compass 743, and a gyroscope 744 may be communicatively coupled tosensor hub 740. In at least one embodiment, thermal sensor 739, a fan737, a keyboard 746, and a touch pad 730 may be communicatively coupledto EC 735. In at least one embodiment, speaker 763, headphones 764, andmicrophone (“mic”) 765 may be communicatively coupled to an audio unit(“audio codec and class d amp”) 762, which may in turn becommunicatively coupled to DSP 760. In at least one embodiment, audiounit 764 may include, for example and without limitation, an audiocoder/decoder (“codec”) and a class D amplifier. In at least oneembodiment, SIM card (“SIM”) 757 may be communicatively coupled to WWANunit 756. In at least one embodiment, components such as WLAN unit 750and Bluetooth unit 752, as well as WWAN unit 756 may be implemented in aNext Generation Form Factor (“NGFF”).

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic x615 may be used in system FIG. 7B forinferencing or predicting operations based, at least in part, on weightparameters calculated using neural network training operations, neuralnetwork functions and/or architectures, or neural network use casesdescribed herein.

FIG. 7C illustrates a computer system 700C, according to at least oneembodiment, to support and/or to enable the intelligent control of amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem as described herein. In at least one embodiment,computer system 700C includes, without limitation, a computer 771 and aUSB stick 770. In at least one embodiment, computer 771 may include,without limitation, any number and type of processor(s) (not shown) anda memory (not shown). In at least one embodiment, computer 771 includes,without limitation, a server, a cloud instance, a laptop, and a desktopcomputer.

In at least one embodiment, USB stick 770 includes, without limitation,a processing unit 772, a USB interface 774, and USB interface logic 773.In at least one embodiment, processing unit 772 may be any instructionexecution system, apparatus, or device capable of executinginstructions. In at least one embodiment, processing unit 772 mayinclude, without limitation, any number and type of processing cores(not shown). In at least one embodiment, processing unit or core 772comprises an application specific integrated circuit (“ASIC”) that isoptimized to perform any amount and type of operations associated withmachine learning. For instance, in at least one embodiment, processingcore 772 is a tensor processing unit (“TPC”) that is optimized toperform machine learning inference operations. In at least oneembodiment, processing core 772 is a vision processing unit (“VPU”) thatis optimized to perform machine vision and machine learning inferenceoperations.

In at least one embodiment, USB interface 774 may be any type of USBconnector or USB socket. For instance, in at least one embodiment, USBinterface 774 is a USB 3.0 Type-C socket for data and power. In at leastone embodiment, USB interface 774 is a USB 3.0 Type-A connector. In atleast one embodiment, USB interface logic 773 may include any amount andtype of logic that enables processing unit 772 to interface with ordevices (such as computer 771) via USB connector 774.

Inference and/or training logic 615, as described with respect to FIGS.6B and 6C, are used to perform inferencing and/or training operationsassociated with one or more embodiments. Details regarding inferenceand/or training logic 615 are provided below in conjunction with FIGS.6B and/or 6C. In at least one embodiment, inference and/or traininglogic 615 may be used in system FIG. 7C for inferencing or predictingoperations based, at least in part, on weight parameters calculatedusing neural network training operations, neural network functionsand/or architectures, or neural network use cases described herein.

FIG. 8 illustrates a further example computer system 800, according toat least one embodiment, to implement various processes and methods forthe repurposable refrigerant cooling subsystem described throughout thisdisclosure. In at least one embodiment, computer system 800 includes,without limitation, at least one central processing unit (“CPU”) 802that is connected to a communication bus 810 implemented using anysuitable protocol, such as PCI (“Peripheral Component Interconnect”),peripheral component interconnect express (“PCI-Express”), AGP(“Accelerated Graphics Port”), HyperTransport, or any other bus orpoint-to-point communication protocol(s). In at least one embodiment,computer system 800 includes, without limitation, a main memory 804 andcontrol logic (such as implemented as hardware, software, or acombination thereof) and data are stored in main memory 804 which maytake form of random access memory (“RAM”). In at least one embodiment, anetwork interface subsystem (“network interface”) 822 provides aninterface to other computing devices and networks for receiving datafrom and transmitting data to other systems from computer system 800.

In at least one embodiment, computer system 800, in at least oneembodiment, includes, without limitation, input devices 808, parallelprocessing system 812, and display devices 806 which can be implementedusing a cathode ray tube (“CRT”), liquid crystal display (“LCD”), lightemitting diode (“LED”), plasma display, or other suitable displaytechnologies. In at least one embodiment, user input is received frominput devices 808 such as keyboard, mouse, touchpad, microphone, andmore. In at least one embodiment, each of foregoing modules can besituated on a single semiconductor platform to form a processing system.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments, aspreviously discussed with respect to FIGS. 6A-C, for instance. Detailsregarding inference and/or training logic 615 are provided below inconjunction with FIGS. 6A-C. In at least one embodiment, inferenceand/or training logic 615 may be used in system FIG. 8 for inferencingor predicting operations based, at least in part, on weight parameterscalculated using neural network training operations, neural networkfunctions and/or architectures, or neural network use cases describedherein. In at least one embodiment, inference and/or training logic 615may be used in system FIG. 8 for inferencing or predicting operationsbased, at least in part, on weight parameters calculated using neuralnetwork training operations, neural network functions and/orarchitectures, or neural network use cases described herein.

FIG. 9A illustrates an exemplary architecture in which a plurality ofGPUs 910-913 is communicatively coupled to a plurality of multi-coreprocessors 905-906 over high-speed links 940-943 (such as buses,point-to-point interconnects, etc.). In one embodiment, high-speed links940-943 support a communication throughput of 4 GB/s, 30 GB/s, 80 GB/sor higher. Various interconnect protocols may be used including, but notlimited to, PCIe 4.0 or 5.0 and NVLink 2.0.

In addition, and in one embodiment, two or more of GPUs 910-913 areinterconnected over high-speed links 929-930, which may be implementedusing same or different protocols/links than those used for high-speedlinks 940-943. Similarly, two or more of multi-core processors 905-906may be connected over high speed link 928 which may be symmetricmulti-processor (SMP) buses operating at 20 GB/s, 30 GB/s, 120 GB/s orhigher. Alternatively, all communication between various systemcomponents shown in FIG. 9A may be accomplished using sameprotocols/links (such as over a common interconnection fabric).

In one embodiment, each multi-core processor 905-906 is communicativelycoupled to a processor memory 901-902, via memory interconnects 926-927,respectively, and each GPU 910-913 is communicatively coupled to GPUmemory 920-923 over GPU memory interconnects 950-953, respectively.Memory interconnects 926-927 and 950-953 may utilize same or differentmemory access technologies. By way of example, and not limitation,processor memories 901-902 and GPU memories 920-923 may be volatilememories such as dynamic random access memories (DRAMs) (includingstacked DRAMs), Graphics DDR SDRAM (GDDR) (such as GDDR5, GDDR6), orHigh Bandwidth Memory (HBM) and/or may be non-volatile memories such as3D XPoint or Nano-Ram. In one embodiment, some portion of processormemories 901-902 may be volatile memory and another portion may benon-volatile memory (such as using a two-level memory (2LM) hierarchy).

As described below, although various processors 905-906 and GPUs 910-913may be physically coupled to a particular memory 901-902, 920-923,respectively, a unified memory architecture may be implemented in whicha same virtual system address space (also referred to as “effectiveaddress” space) is distributed among various physical memories. In atleast one embodiment, processor memories 901-902 may each include 64 GBof system memory address space and GPU memories 920-923 may each include32 GB of system memory address space (resulting in a total of 256 GBaddressable memory in this example).

As discussed elsewhere in this disclosure, at least flow rates andassociated temperatures may be established for a first level of anintelligent learning system, such as a neural network system. As thefirst level represents the prior data, it also represents a smallersubset of the data that may be available to improve the system byretraining the system. The testing and training may be performed inparallel using the multiple processor units so that the intelligentlearning system is robust. An architecture, such as in FIG. 9A, may beused. When convergence is achieved for the intelligent learning system,an amount of data points and the data in the data points used to causethe convergence is noted. The data and data points may be used tocontrol the repurposable refrigerant cooling subsystem and the multiplemode cooling subsystem, entirely, as discussed in reference, forinstance, to FIGS. 2A-5.

FIG. 9B illustrates additional details for an interconnection between amulti-core processor 907 and a graphics acceleration module 946 inaccordance with one exemplary embodiment. Graphics acceleration module946 may include one or more GPU chips integrated on a line card which iscoupled to processor 907 via high-speed link 940. Alternatively,graphics acceleration module 946 may be integrated on a same package orchip as processor 907.

In at least one embodiment, illustrated processor 907 includes aplurality of cores 960A-960D, each with a translation lookaside buffer961A-961D and one or more caches 962A-962D. In at least one embodiment,cores 960A-960D may include various other components for executinginstructions and processing data which are not illustrated. Caches962A-962D may include level 1 (L1) and level 2 (L2) caches. In addition,one or more shared caches 956 may be included in caches 962A-962D andshared by sets of cores 960A-960D. In at least one embodiment, oneembodiment of processor 907 includes 24 cores, each with its own L1cache, twelve shared L2 caches, and twelve shared L3 caches. In thisembodiment, one or more L2 and L3 caches are shared by two adjacentcores. Processor 907 and graphics acceleration module 946 connect withsystem memory 914, which may include processor memories 901-902 of FIG.9A.

Coherency is maintained for data and instructions stored in variouscaches 962A-962D, 956 and system memory 914 via inter-core communicationover a coherence bus 964. In at least one embodiment, each cache mayhave cache coherency logic/circuitry associated therewith to communicateto over coherence bus 964 in response to detected reads or writes toparticular cache lines. In one implementation, a cache snooping protocolis implemented over coherence bus 964 to snoop cache accesses.

In one embodiment, a proxy circuit 925 communicatively couples graphicsacceleration module 946 to coherence bus 964, allowing graphicsacceleration module 946 to participate in a cache coherence protocol asa peer of cores 960A-960D. In particular, an interface 935 providesconnectivity to proxy circuit 925 over high-speed link 940 (such as aPCIe bus, NVLink, etc.) and an interface 937 connects graphicsacceleration module 946 to link 940.

In one implementation, an accelerator integration circuit 936 providescache management, memory access, context management, and interruptmanagement services on behalf of a plurality of graphics processingengines 931, 932, N of graphics acceleration module 946. Graphicsprocessing engines 931, 932, N may each include a separate graphicsprocessing unit (GPU). Alternatively, graphics processing engines 931,932, N may include different types of graphics processing engines withina GPU such as graphics execution units, media processing engines (suchas video encoders/decoders), samplers, and blit engines. In at least oneembodiment, graphics acceleration module 946 may be a GPU with aplurality of graphics processing engines 931-932, N or graphicsprocessing engines 931-932, N may be individual GPUs integrated on acommon package, line card, or chip. As is the case, the abovedetermination for the reconstruction parameter and the reconstructionalgorithm may be performed in GPUs 931-N of FIG. 9B.

In one embodiment, accelerator integration circuit 936 includes a memorymanagement unit (MMU) 939 for performing various memory managementfunctions such as virtual-to-physical memory translations (also referredto as effective-to-real memory translations) and memory access protocolsfor accessing system memory 914. MMU 939 may also include a translationlookaside buffer (TLB) (not shown) for caching virtual/effective tophysical/real address translations. In one implementation, a cache 938stores commands and data for efficient access by graphics processingengines 931-932, N. In one embodiment, data stored in cache 938 andgraphics memories 933-934, M is kept coherent with core caches962A-962D, 956, and system memory 914. As mentioned above, this may beaccomplished via proxy circuit 925 on behalf of cache 938 and memories933-934, M (such as sending updates to cache 938 related tomodifications/accesses of cache lines on processor caches 962A-962D,956, and receiving updates from cache 938).

A set of registers 945 store context data for threads executed bygraphics processing engines 931-932, N and a context management circuit948 manages thread contexts. In at least one embodiment, contextmanagement circuit 948 may perform save and restore operations to saveand restore contexts of various threads during contexts switches (suchas where a first thread is saved and a second thread is stored so that asecond thread can be executed by a graphics processing engine). In atleast one embodiment, on a context switch, context management circuit948 may store current register values to a designated region in memory(such as identified by a context pointer). It may then restore registervalues when returning to a context. In one embodiment, an interruptmanagement circuit 947 receives and processes interrupts received fromsystem devices.

In one implementation, virtual/effective addresses from a graphicsprocessing engine 931 are translated to real/physical addresses insystem memory 914 by MMU 939. One embodiment of accelerator integrationcircuit 936 supports multiple (such as 4, 8, 16) graphics acceleratormodules 946 and/or other accelerator devices. Graphics acceleratormodule 946 may be dedicated to a single application executed onprocessor 907 or may be shared between multiple applications. In oneembodiment, a virtualized graphics execution environment is presented inwhich resources of graphics processing engines 931-932, N are sharedwith multiple applications or virtual machines (VMs). In at least oneembodiment, resources may be subdivided into “slices” which areallocated to different VMs and/or applications based on processingrequirements and priorities associated with VMs and/or applications.

In at least one embodiment, accelerator integration circuit 936 performsas a bridge to a system for graphics acceleration module 946 andprovides address translation and system memory cache services. Inaddition, accelerator integration circuit 936 may provide virtualizationfacilities for a host processor to manage virtualization of graphicsprocessing engines 931-932, N, interrupts, and memory management.

Because hardware resources of graphics processing engines 931-932, N aremapped explicitly to a real address space seen by host processor 907,any host processor can address these resources directly using aneffective address value. One function of accelerator integration circuit936, in one embodiment, is physical separation of graphics processingengines 931-932, N so that they appear to a system as independent units.

In at least one embodiment, one or more graphics memories 933-934, M arecoupled to each of graphics processing engines 931-932, N, respectively.Graphics memories 933-934, M store instructions and data being processedby each of graphics processing engines 931-932, N. Graphics memories933-934, M may be volatile memories such as DRAMs (including stackedDRAMs), GDDR memory (such as GDDR5, GDDR6), or HBM, and/or may benon-volatile memories such as 3D XPoint or Nano-Ram.

In one embodiment, to reduce data traffic over link 940, biasingtechniques are used to ensure that data stored in graphics memories933-934, M is data which will be used most frequently by graphicsprocessing engines 931-932, N and may not used by cores 960A-960D (atleast not frequently). Similarly, a biasing mechanism attempts to keepdata needed by cores (and may not graphics processing engines 931-932,N) within caches 962A-962D, 956 of cores and system memory 914.

FIG. 9C illustrates another exemplary embodiment in which acceleratorintegration circuit 936 is integrated within processor 907 for enablingand/or supporting intelligent control of a multiple mode coolingsubsystem having a repurposable refrigerant cooling subsystem, accordingto at least one embodiment of the disclosure herein. In at least thisembodiment, graphics processing engines 931-932, N communicate directlyover high-speed link 940 to accelerator integration circuit 936 viainterface 937 and interface 935 (which, again, may be utilize any formof bus or interface protocol). Accelerator integration circuit 936 mayperform same operations as those described with respect to FIG. 9B, butpotentially at a higher throughput given its close proximity tocoherence bus 964 and caches 962A-962D, 956. At least one embodimentsupports different programming models including a dedicated-processprogramming model (no graphics acceleration module virtualization) andshared programming models (with virtualization), which may includeprogramming models which are controlled by accelerator integrationcircuit 936 and programming models which are controlled by graphicsacceleration module 946.

In at least one embodiment, graphics processing engines 931-932, N arededicated to a single application or process under a single operatingsystem. In at least one embodiment, a single application can funnelother application requests to graphics processing engines 931-932, N,providing virtualization within a VM/partition.

In at least one embodiment, graphics processing engines 931-932, N, maybe shared by multiple VM/application partitions. In at least oneembodiment, shared models may use a system hypervisor to virtualizegraphics processing engines 931-932, N to allow access by each operatingsystem. For single-partition systems without a hypervisor, graphicsprocessing engines 931-932, N are owned by an operating system. In atleast one embodiment, an operating system can virtualize graphicsprocessing engines 931-932, N to provide access to each process orapplication.

In at least one embodiment, graphics acceleration module 946 or anindividual graphics processing engine 931-932, N selects a processelement using a process handle. In at least one embodiment, processelements are stored in system memory 914 and are addressable using aneffective address to real address translation techniques describedherein. In at least one embodiment, a process handle may be animplementation-specific value provided to a host process whenregistering its context with graphics processing engine 931-932, N (thatis, calling system software to add a process element to a processelement linked list). In at least one embodiment, a lower 16-bits of aprocess handle may be an offset of a process element within a processelement linked list.

FIG. 9D illustrates an exemplary accelerator integration slice 990 forenabling and/or supporting intelligent control of a multiple modecooling subsystem having a repurposable refrigerant cooling subsystem,according to at least one embodiment of the disclosure herein. As usedherein, a “slice” comprises a specified portion of processing resourcesof accelerator integration circuit 936. Application effective addressspace 982 within system memory 914 stores process elements 983. In oneembodiment, process elements 983 are stored in response to GPUinvocations 981 from applications 980 executed on processor 907. Aprocess element 983 contains process state for corresponding application980. A work descriptor (WD) 984 contained in process element 983 can bea single job requested by an application or may contain a pointer to aqueue of jobs. In at least one embodiment, WD 984 is a pointer to a jobrequest queue in an application's address space 982.

Graphics acceleration module 946 and/or individual graphics processingengines 931-932, N can be shared by all or a subset of processes in asystem. In at least one embodiment, an infrastructure for setting upprocess state and sending a WD 984 to a graphics acceleration module 946to start a job in a virtualized environment may be included.

In at least one embodiment, a dedicated-process programming model isimplementation-specific. In this model, a single process owns graphicsacceleration module 946 or an individual graphics processing engine 931.Because graphics acceleration module 946 is owned by a single process, ahypervisor initializes accelerator integration circuit 936 for an owningpartition and an operating system initializes accelerator integrationcircuit 936 for an owning process when graphics acceleration module 946is assigned.

In operation, a WD fetch unit 991 in accelerator integration slice 990fetches next WD 984 which includes an indication of work to be done byone or more graphics processing engines of graphics acceleration module946. Data from WD 984 may be stored in registers 945 and used by MMU939, interrupt management circuit 947, and/or context management circuit948 as illustrated. In at least one embodiment, one embodiment of MMU939 includes segment/page walk circuitry for accessing segment/pagetables 986 within OS virtual address space 985. Interrupt managementcircuit 947 may process interrupt events 992 received from graphicsacceleration module 946. When performing graphics operations, aneffective address 993 generated by a graphics processing engine 931-932,N is translated to a real address by MMU 939.

In one embodiment, a same set of registers 945 are duplicated for eachgraphics processing engine 931-932, N and/or graphics accelerationmodule 946 and may be initialized by a hypervisor or operating system.Each of these duplicated registers may be included in an acceleratorintegration slice 990. Exemplary registers that may be initialized by ahypervisor are shown in Table 1.

TABLE 1 Hypervisor Initialized Registers 1 Slice Control Register 2 RealAddress (RA) Scheduled Processes Area Pointer 3 Authority Mask OverrideRegister 4 Interrupt Vector Table Entry Offset 5 Interrupt Vector TableEntry Limit 6 State Register 7 Logical Partition ID 8 Real address (RA)Hypervisor Accelerator Utilization Record Pointer 9 Storage DescriptionRegister

Exemplary registers that may be initialized by an operating system areshown in Table 2.

TABLE 2 Operating System Initialized Registers 1 Process and ThreadIdentification 2 Effective Address (EA) Context Save/Restore Pointer 3Virtual Address (VA) Accelerator Utilization Record Pointer 4 VirtualAddress (VA) Storage Segment Table Pointer 5 Authority Mask 6 Workdescriptor

In one embodiment, each WD 984 is specific to a particular graphicsacceleration module 946 and/or graphics processing engines 931-932, N.It contains all information required by a graphics processing engine931-932, N to do work or it can be a pointer to a memory location wherean application has set up a command queue of work to be completed.

FIG. 9E illustrates additional details for one exemplary embodiment of ashared model. This embodiment includes a hypervisor real address space998 in which a process element list 999 is stored. Hypervisor realaddress space 998 is accessible via a hypervisor 996 which virtualizesgraphics acceleration module engines for operating system 995.

In at least one embodiment, shared programming models allow for all or asubset of processes from all or a subset of partitions in a system touse a graphics acceleration module 946. There are two programming modelswhere graphics acceleration module 946 is shared by multiple processesand partitions: time-sliced shared and graphics-directed shared.

In this model, system hypervisor 996 owns graphics acceleration module946 and makes its function available to all operating systems 995. For agraphics acceleration module 946 to support virtualization by systemhypervisor 996, graphics acceleration module 946 may adhere to thefollowing: 1) An application's job request must be autonomous (that is,state does not need to be maintained between jobs), or graphicsacceleration module 946 must provide a context save and restoremechanism. 2) An application's job request is guaranteed by graphicsacceleration module 946 to complete in a specified amount of time,including any translation faults, or graphics acceleration module 946provides an ability to preempt processing of a job. 3) Graphicsacceleration module 946 must be guaranteed fairness between processeswhen operating in a directed shared programming model.

In at least one embodiment, application 980 is required to make anoperating system 995 system call with a graphics acceleration module 946type, a work descriptor (WD), an authority mask register (AMR) value,and a context save/restore area pointer (CSRP). In at least oneembodiment, graphics acceleration module 946 type describes a targetedacceleration function for a system call. In at least one embodiment,graphics acceleration module 946 type may be a system-specific value. Inat least one embodiment, WD is formatted specifically for graphicsacceleration module 946 and can be in a form of a graphics accelerationmodule 946 command, an effective address pointer to a user-definedstructure, an effective address pointer to a queue of commands, or anyother data structure to describe work to be done by graphicsacceleration module 946. In one embodiment, an AMR value is an AMR stateto use for a current process. In at least one embodiment, a value passedto an operating system is similar to an application setting an AMR. Ifaccelerator integration circuit 936 and graphics acceleration module 946implementations do not support a User Authority Mask Override Register(UAMOR), an operating system may apply a current UAMOR value to an AMRvalue before passing an AMR in a hypervisor call. Hypervisor 996 may, inat least one embodiment, apply a current Authority Mask OverrideRegister (AMOR) value before placing an AMR into process element 983. Inat least one embodiment, CSRP is one of registers 945 containing aneffective address of an area in an application's effective address space982 for graphics acceleration module 946 to save and restore contextstate. This pointer is used in at least one embodiment, if no state isrequired to be saved between jobs or when a job is preempted. In atleast one embodiment, context save/restore area may be pinned systemmemory.

Upon receiving a system call, operating system 995 may verify thatapplication 980 has registered and been given authority to use graphicsacceleration module 946. Operating system 995 then calls hypervisor 996with information shown in Table 3.

TABLE 3 OS to Hypervisor Call Parameters 1 A work descriptor (WD) 2 AnAuthority Mask Register (AMR) value (potentially masked) 3 An effectiveaddress (EA) Context Save/Restore Area Pointer (CSRP) 4 A process ID(PID) and optional thread ID (TID) 5 A virtual address (VA) acceleratorutilization record pointer (AURP) 6 Virtual address of storage segmenttable pointer (SSTP) 7 A logical interrupt service number (LISN)

Upon receiving a hypervisor call, hypervisor 996 verifies that operatingsystem 995 has registered and been given authority to use graphicsacceleration module 946. Hypervisor 996 then puts process element 983into a process element linked list for a corresponding graphicsacceleration module 946 type. A process element may include informationshown in Table 4.

TABLE 4 Process Element Information 1 A work descriptor (WD) 2 AnAuthority Mask Register (AMR) value (potentially masked). 3 An effectiveaddress (EA) Context Save/Restore Area Pointer (CSRP) 4 A process ID(PID) and optional thread ID (TID) 5 A virtual address (VA) acceleratorutilization record pointer (AURP) 6 Virtual address of storage segmenttable pointer (SSTP) 7 A logical interrupt service number (LISN) 8Interrupt vector table, derived from hypervisor call parameters 9 Astate register (SR) value 10 A logical partition ID (LPID) 11 A realaddress (RA) hypervisor accelerator utilization record pointer 12Storage Descriptor Register (SDR)

In at least one embodiment, hypervisor initializes a plurality ofaccelerator integration slice 990 registers 945.

As illustrated in FIG. 9F, in at least one embodiment, a unified memoryis used, addressable via a common virtual memory address space used toaccess physical processor memories 901-902 and GPU memories 920-923. Inthis implementation, operations executed on GPUs 910-913 utilize a samevirtual/effective memory address space to access processor memories901-902 and vice versa, thereby simplifying programmability. In oneembodiment, a first portion of a virtual/effective address space isallocated to processor memory 901, a second portion to second processormemory 902, a third portion to GPU memory 920, and so on. In at leastone embodiment, an entire virtual/effective memory space (sometimesreferred to as an effective address space) is thereby distributed acrosseach of processor memories 901-902 and GPU memories 920-923, allowingany processor or GPU to access any physical memory with a virtualaddress mapped to that memory.

In one embodiment, bias/coherence management circuitry 994A-994E withinone or more of MMUs 939A-939E ensures cache coherence between caches ofone or more host processors (such as 905) and GPUs 910-913 andimplements biasing techniques indicating physical memories in whichcertain types of data should be stored. While multiple instances ofbias/coherence management circuitry 994A-994E are illustrated in FIG.9F, bias/coherence circuitry may be implemented within an MMU of one ormore host processors 905 and/or within accelerator integration circuit936.

One embodiment allows GPU-attached memory 920-923 to be mapped as partof system memory, and accessed using shared virtual memory (SVM)technology, but without suffering performance drawbacks associated withfull system cache coherence. In at least one embodiment, an ability forGPU-attached memory 920-923 to be accessed as system memory withoutonerous cache coherence overhead provides a beneficial operatingenvironment for GPU offload. This arrangement allows host processor 905software to setup operands and access computation results, withoutoverhead of tradition I/O DMA data copies. Such traditional copiesinvolve driver calls, interrupts and memory mapped I/O (MMIO) accessesthat are all inefficient relative to simple memory accesses. In at leastone embodiment, an ability to access GPU attached memory 920-923 withoutcache coherence overheads can be critical to execution time of anoffloaded computation. In cases with substantial streaming write memorytraffic, for example, cache coherence overhead can significantly reducean effective write bandwidth seen by a GPU 910-913. In at least oneembodiment, efficiency of operand setup, efficiency of results access,and efficiency of GPU computation may play a role in determiningeffectiveness of a GPU offload.

In at least one embodiment, selection of GPU bias and host processorbias is driven by a bias tracker data structure. A bias table may beused, for example, which may be a page-granular structure (in at leastone embodiment this may be controlled at a granularity of a memory page)that includes 1 or 2 bits per GPU-attached memory page. In at least oneembodiment, a bias table may be implemented in a stolen memory range ofone or more GPU-attached memories 920-923, with or without a bias cachein GPU 910-913 (such as to cache frequently/recently used entries of abias table). Alternatively, an entire bias table may be maintainedwithin a GPU.

In at least one embodiment, a bias table entry associated with eachaccess to GPU-attached memory 920-923 is accessed prior to actual accessto a GPU memory, causing the following operations. First, local requestsfrom GPU 910-913 that find their page in GPU bias are forwarded directlyto a corresponding GPU memory 920-923. Local requests from a GPU thatfind their page in host bias are forwarded to processor 905 (such asover a high-speed link as discussed above). In one embodiment, requestsfrom processor 905 that find a requested page in host processor biascomplete a request like a normal memory read. Alternatively, requestsdirected to a GPU-biased page may be forwarded to GPU 910-913. In atleast one embodiment, a GPU may then transition a page to a hostprocessor bias if it is not currently using a page. In at least oneembodiment, bias state of a page can be changed either by asoftware-based mechanism, a hardware-assisted software-based mechanism,or, for a limited set of cases, a purely hardware-based mechanism.

One mechanism for changing bias state employs an API call (such asOpenCL), which, in turn, calls a GPU's device driver which, in turn,sends a message (or enqueues a command descriptor) to a GPU directing itto change a bias state and, for some transitions, perform a cacheflushing operation in a host. In at least one embodiment, cache flushingoperation is used for a transition from host processor 905 bias to GPUbias, but is not for an opposite transition.

In one embodiment, cache coherency is maintained by temporarilyrendering GPU-biased pages uncacheable by host processor 905. To accessthese pages, processor 905 may request access from GPU 910 which may ormay not grant access right away. Thus, to reduce communication betweenprocessor 905 and GPU 910 it is beneficial to ensure that GPU-biasedpages are those which are required by a GPU but not host processor 905and vice versa.

Inference and/or training logic 615 are used to perform one or moreembodiments. Details regarding the inference and/or training logic 615are provided below in conjunction with FIGS. 6B and/or 6C.

FIG. 10A illustrates exemplary integrated circuits and associatedgraphics processors that may be fabricated using one or more IP cores,according to various embodiments described herein, to support and/or toenable a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem as described herein. In addition to whatis illustrated, other logic and circuits may be included in at least oneembodiment, including additional graphics processors/cores, peripheralinterface controllers, or general-purpose processor cores.

FIG. 10A is a block diagram illustrating an exemplary system on a chipintegrated circuit 1000A that may be fabricated using one or more IPcores, according to at least one embodiment. In at least one embodiment,integrated circuit 1000A includes one or more application processor(s)1005 (such as CPUs), at least one graphics processor 1010, and mayadditionally include an image processor 1015 and/or a video processor1020, any of which may be a modular IP core. In at least one embodiment,integrated circuit 1000A includes peripheral or bus logic including aUSB controller 1025, UART controller 1030, an SPI/SDIO controller 1035,and an I²S/I²C controller 1040. In at least one embodiment, integratedcircuit 1000A can include a display device 1045 coupled to one or moreof a high-definition multimedia interface (HDMI) controller 1050 and amobile industry processor interface (MIPI) display interface 1055. In atleast one embodiment, storage may be provided by a flash memorysubsystem 1060 including flash memory and a flash memory controller. Inat least one embodiment, memory interface may be provided via a memorycontroller 1065 for access to SDRAM or SRAM memory devices. In at leastone embodiment, some integrated circuits additionally include anembedded security engine 1070.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in integrated circuit1000A for inferencing or predicting operations based, at least in part,on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIGS. 10B-10C illustrate exemplary integrated circuits and associatedgraphics processors that may be fabricated using one or more IP cores,according to various embodiments described herein to support and/or toenable a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem as described herein. In addition to whatis illustrated, other logic and circuits may be included in at least oneembodiment, including additional graphics processors/cores, peripheralinterface controllers, or general-purpose processor cores.

FIGS. 10B-10C are block diagrams illustrating exemplary graphicsprocessors for use within an SoC, according to embodiments describedherein, to support and/or to enable a multiple mode cooling subsystemhaving a repurposable refrigerant cooling subsystem as described herein.In an example, the graphic processors may be used in the intelligentcontrol of a multiple mode cooling subsystem having a repurposablerefrigerant cooling subsystem because of existing math engines capableof faster processing of multi-level neural networks. FIG. 10Billustrates an exemplary graphics processor 1010 of a system on a chipintegrated circuit that may be fabricated using one or more IP cores,according to at least one embodiment. FIG. 10C illustrates an additionalexemplary graphics processor 1040 of a system on a chip integratedcircuit that may be fabricated using one or more IP cores, according toat least one embodiment. In at least one embodiment, graphics processor1010 of FIG. 10A is a low power graphics processor core. In at least oneembodiment, graphics processor 1040 of FIG. 10C is a higher performancegraphics processor core. In at least one embodiment, each of graphicsprocessors 1010, 1040 can be variants of graphics processor 1010 of FIG.10A.

In at least one embodiment, graphics processor 1010 includes a vertexprocessor 1005 and one or more fragment processor(s) 1015A-1015N (suchas 1015A, 1015B, 1015C, 1015D, through 1015N-1, and 1015N). In at leastone embodiment, graphics processor 1010 can execute different shaderprograms via separate logic, such that vertex processor 1005 isoptimized to execute operations for vertex shader programs, while one ormore fragment processor(s) 1015A-1015N execute fragment (such as pixel)shading operations for fragment or pixel shader programs. In at leastone embodiment, vertex processor 1005 performs a vertex processing stageof a 3D graphics pipeline and generates primitives and vertex data. Inat least one embodiment, fragment processor(s) 1015A-1015N use primitiveand vertex data generated by vertex processor 1005 to produce aframebuffer that is displayed on a display device. In at least oneembodiment, fragment processor(s) 1015A-1015N are optimized to executefragment shader programs as provided for in an OpenGL API, which may beused to perform similar operations as a pixel shader program as providedfor in a Direct 3D API.

In at least one embodiment, graphics processor 1010 additionallyincludes one or more memory management units (MMUs) 1020A-1020B,cache(s) 1025A-1025B, and circuit interconnect(s) 1030A-1030B. In atleast one embodiment, one or more MMU(s) 1020A-1020B provide for virtualto physical address mapping for graphics processor 1010, including forvertex processor 1005 and/or fragment processor(s) 1015A-1015N, whichmay reference vertex or image/texture data stored in memory, in additionto vertex or image/texture data stored in one or more cache(s)1025A-1025B. In at least one embodiment, one or more MMU(s) 1020A-1020Bmay be synchronized with other MMUs within system, including one or moreMMUs associated with one or more application processor(s) 1005, imageprocessors 1015, and/or video processors 1020 of FIG. 10A, such thateach processor 1005-1020 can participate in a shared or unified virtualmemory system. In at least one embodiment, one or more circuitinterconnect(s) 1030A-1030B enable graphics processor 1010 to interfacewith other IP cores within SoC, either via an internal bus of SoC or viaa direct connection.

In at least one embodiment, graphics processor 1040 includes one or moreMMU(s) 1020A-1020B, cache(s) 1025A-1025B, and circuit interconnect(s)1030A-1030B of graphics processor 1010 of FIG. 10A. In at least oneembodiment, graphics processor 1040 includes one or more shader core(s)1055A-1055N (such as 1055A, 1055B, 1055C, 1055D, 1055E, 1055F, through1055N-1, and 1055N), which provides for a unified shader corearchitecture in which a single core or type or core can execute alltypes of programmable shader code, including shader program code toimplement vertex shaders, fragment shaders, and/or compute shaders. Inat least one embodiment, a number of shader cores can vary. In at leastone embodiment, graphics processor 1040 includes an inter-core taskmanager 1045, which acts as a thread dispatcher to dispatch executionthreads to one or more shader cores 1055A-1055N and a tiling unit 1058to accelerate tiling operations for tile-based rendering, in whichrendering operations for a scene are subdivided in image space, forexample to exploit local spatial coherence within a scene or to optimizeuse of internal caches.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in integrated circuit10A and/or 10B for inferencing or predicting operations based, at leastin part, on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIGS. 10D-10E illustrate additional exemplary graphics processor logicaccording to embodiments described herein to support and/or to enable amultiple mode cooling subsystem having a repurposable refrigerantcooling subsystem as described herein. FIG. 10D illustrates a graphicscore 1000D that may be included within graphics processor 1010 of FIG.10A, in at least one embodiment, and may be a unified shader core1055A-1055N as in FIG. 10C in at least one embodiment. FIG. 10Billustrates a highly-parallel general-purpose graphics processing unit1030 suitable for deployment on a multi-chip module in at least oneembodiment.

In at least one embodiment, graphics core 1000D can include multipleslices 1001A-1001N or partition for each core, and a graphics processorcan include multiple instances of graphics core 1000D. Slices1001A-1001N can include support logic including a local instructioncache 1004A-1004N, a thread scheduler 1006A-1006N, a thread dispatcher1008A-1008N, and a set of registers 1010A-1010N. In at least oneembodiment, slices 1001A-1001N can include a set of additional functionunits (AFUs 1012A-1012N), floating-point units (FPU 1014A-1014N),integer arithmetic logic units (ALUs 1016-1016N), address computationalunits (ACU 1013A-1013N), double-precision floating-point units (DPFPU1015A-1015N), and matrix processing units (MPU 1017A-1017N).

In at least one embodiment, FPUs 1014A-1014N can performsingle-precision (32-bit) and half-precision (16-bit) floating pointoperations, while DPFPUs 1015A-1015N perform double precision (64-bit)floating point operations. In at least one embodiment, ALUs 1016A-1016Ncan perform variable precision integer operations at 8-bit, 16-bit, and32-bit precision, and can be configured for mixed precision operations.In at least one embodiment, MPUs 1017A-1017N can also be configured formixed precision matrix operations, including half-precision floatingpoint and 8-bit integer operations. In at least one embodiment, MPUs1017A-1017N can perform a variety of matrix operations to acceleratemachine learning application frameworks, including enabling support foraccelerated general matrix to matrix multiplication (GEMM). In at leastone embodiment, AFUs 1012A-1012N can perform additional logic operationsnot supported by floating-point or integer units, includingtrigonometric operations (such as Sine, Cosine, etc.).

As discussed elsewhere in this disclosure, inference and/or traininglogic 615 (referenced at least in FIGS. 6B, 6C) may be used to performinferencing and/or training operations associated with one or moreembodiments. Details regarding inference and/or training logic 615 areprovided below in conjunction with FIGS. 6B and/or 6C. In at least oneembodiment, inference and/or training logic 615 may be used in graphicscore 1000D for inferencing or predicting operations based, at least inpart, on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIG. 11A is a block diagram illustrating a computing system 1100Aaccording to at least one embodiment. In at least one embodiment,computing system 1100A includes a processing subsystem 1101 having oneor more processor(s) 1102 and a system memory 1104 communicating via aninterconnection path that may include a memory hub 1105. In at least oneembodiment, memory hub 1105 may be a separate component within a chipsetcomponent or may be integrated within one or more processor(s) 1102. Inat least one embodiment, memory hub 1105 couples with an I/O subsystem1111 via a communication link 1106. In at least one embodiment, I/Osubsystem 1111 includes an I/O hub 1107 that can enable computing system1100A to receive input from one or more input device(s) 1108. In atleast one embodiment, I/O hub 1107 can enable a display controller,which may be included in one or more processor(s) 1102, to provideoutputs to one or more display device(s) 1110A. In at least oneembodiment, one or more display device(s) 1110A coupled with I/O hub1107 can include a local, internal, or embedded display device.

In at least one embodiment, processing subsystem 1101 includes one ormore parallel processor(s) 1112 coupled to memory hub 1105 via a bus orother communication link 1113. In at least one embodiment, communicationlink 1113 may be one of any number of standards based communication linktechnologies or protocols, such as but not limited to PCI Express, ormay be a vendor specific communications interface or communicationsfabric. In at least one embodiment, one or more parallel processor(s)1112 form a computationally focused parallel or vector processing systemthat can include a large number of processing cores and/or processingclusters, such as a many integrated core (MIC) processor. In at leastone embodiment, one or more parallel processor(s) 1112 form a graphicsprocessing subsystem that can output pixels to one of one or moredisplay device(s) 1110A coupled via I/O Hub 1107. In at least oneembodiment, one or more parallel processor(s) 1112 can also include adisplay controller and display interface (not shown) to enable a directconnection to one or more display device(s) 1110B.

In at least one embodiment, a system storage unit 1114 can connect toI/O hub 1107 to provide a storage mechanism for computing system 1100A.In at least one embodiment, an I/O switch 1116 can be used to provide aninterface mechanism to enable connections between I/O hub 1107 and othercomponents, such as a network adapter 1118 and/or wireless networkadapter 1119 that may be integrated into a platform(s), and variousother devices that can be added via one or more add-in device(s) 1120.In at least one embodiment, network adapter 1118 can be an Ethernetadapter or another wired network adapter. In at least one embodiment,wireless network adapter 1119 can include one or more of a Wi-Fi,Bluetooth, near field communication (NFC), or other network device thatincludes one or more wireless radios.

In at least one embodiment, computing system 1100A can include othercomponents not explicitly shown, including USB or other portconnections, optical storage drives, video capture devices, and so on,may also be connected to I/O hub 1107. In at least one embodiment,communication paths interconnecting various components in FIG. 11A maybe implemented using any suitable protocols, such as PCI (PeripheralComponent Interconnect) based protocols (such as PCI-Express), or otherbus or point-to-point communication interfaces and/or protocol(s), suchas NV-Link high-speed interconnect, or interconnect protocols.

In at least one embodiment, one or more parallel processor(s) 1112incorporate circuitry optimized for graphics and video processing,including, for example, video output circuitry, and constitutes agraphics processing unit (GPU). In at least one embodiment, one or moreparallel processor(s) 1112 incorporate circuitry optimized for generalpurpose processing. In at least one embodiment, components of computingsystem 1100A may be integrated with one or more other system elements ona single integrated circuit. In at least one embodiment, in at least oneembodiment, one or more parallel processor(s) 1112, memory hub 1105,processor(s) 1102, and I/O hub 1107 can be integrated into a system onchip (SoC) integrated circuit. In at least one embodiment, components ofcomputing system 1100A can be integrated into a single package to form asystem in package (SIP) configuration. In at least one embodiment, atleast a portion of components of computing system 1100A can beintegrated into a multi-chip module (MCM), which can be interconnectedwith other multi-chip modules into a modular computing system.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in system FIG. 11A forinferencing or predicting operations based, at least in part, on weightparameters calculated using neural network training operations, neuralnetwork functions and/or architectures, or neural network use casesdescribed herein.

Processors

FIG. 11B illustrates a parallel processor 1100B according to at leastone embodiment. In at least one embodiment, various components ofparallel processor 1100B may be implemented using one or more integratedcircuit devices, such as programmable processors, application specificintegrated circuits (ASICs), or field programmable gate arrays (FPGA).In at least one embodiment, illustrated parallel processor 11100B 100 isa variant of one or more parallel processor(s) 1112 shown in FIG. 11Baccording to an exemplary embodiment.

In at least one embodiment, parallel processor 1100B includes a parallelprocessing unit 1102. In at least one embodiment, parallel processingunit 1102 includes an I/O unit 1104 that enables communication withother devices, including other instances of parallel processing unit1102. In at least one embodiment, I/O unit 1104 may be directlyconnected to other devices. In at least one embodiment, I/O unit 1104connects with other devices via use of a hub or switch interface, suchas memory hub 1105. In at least one embodiment, connections betweenmemory hub 1105 and I/O unit 1104 form a communication link 1113. In atleast one embodiment, I/O unit 1104 connects with a host interface 1106and a memory crossbar 1116, where host interface 1106 receives commandsdirected to performing processing operations and memory crossbar 1116receives commands directed to performing memory operations.

In at least one embodiment, when host interface 1106 receives a commandbuffer via I/O unit 1104, host interface 1106 can direct work operationsto perform those commands to a front end 1108. In at least oneembodiment, front end 1108 couples with a scheduler 1110, which isconfigured to distribute commands or other work items to a processingcluster array 1112. In at least one embodiment, scheduler 1110 ensuresthat processing cluster array 1112 is properly configured and in a validstate before tasks are distributed to processing cluster array 1112. Inat least one embodiment, scheduler 1110 is implemented via firmwarelogic executing on a microcontroller. In at least one embodiment,microcontroller implemented scheduler 1110 is configurable to performcomplex scheduling and work distribution operations at coarse and finegranularity, enabling rapid preemption and context switching of threadsexecuting on processing array 1112. In at least one embodiment, hostsoftware can prove workloads for scheduling on processing array 1112 viaone of multiple graphics processing doorbells. In at least oneembodiment, workloads can then be automatically distributed acrossprocessing array 1112 by scheduler 1110 logic within a microcontrollerincluding scheduler 1110.

In at least one embodiment, processing cluster array 1112 can include upto “N” processing clusters (such as cluster 1114A, cluster 1114B,through cluster 1114N). In at least one embodiment, each cluster1114A-1114N of processing cluster array 1112 can execute a large numberof concurrent threads. In at least one embodiment, scheduler 1110 canallocate work to clusters 1114A-1114N of processing cluster array 1112using various scheduling and/or work distribution algorithms, which mayvary depending on workload arising for each type of program orcomputation. In at least one embodiment, scheduling can be handleddynamically by scheduler 1110, or can be assisted in part by compilerlogic during compilation of program logic configured for execution byprocessing cluster array 1112. In at least one embodiment, differentclusters 1114A-1114N of processing cluster array 1112 can be allocatedfor processing different types of programs or for performing differenttypes of computations.

In at least one embodiment, processing cluster array 1112 can beconfigured to perform various types of parallel processing operations.In at least one embodiment, processing cluster array 1112 is configuredto perform general-purpose parallel compute operations. In at least oneembodiment, in at least one embodiment, processing cluster array 1112can include logic to execute processing tasks including filtering ofvideo and/or audio data, performing modeling operations, includingphysics operations, and performing data transformations.

In at least one embodiment, processing cluster array 1112 is configuredto perform parallel graphics processing operations. In at least oneembodiment, processing cluster array 1112 can include additional logicto support execution of such graphics processing operations, including,but not limited to texture sampling logic to perform texture operations,as well as tessellation logic and other vertex processing logic. In atleast one embodiment, processing cluster array 1112 can be configured toexecute graphics processing related shader programs such as but notlimited to vertex shaders, tessellation shaders, geometry shaders, andpixel shaders. In at least one embodiment, parallel processing unit 1102can transfer data from system memory via I/O unit 1104 for processing.In at least one embodiment, during processing, transferred data can bestored to on-chip memory (such as parallel processor memory 1122) duringprocessing, then written back to system memory.

In at least one embodiment, when parallel processing unit 1102 is usedto perform graphics processing, scheduler 1110 can be configured todivide a processing workload into approximately equal sized tasks, tobetter enable distribution of graphics processing operations to multipleclusters 1114A-1114N of processing cluster array 1112. In at least oneembodiment, portions of processing cluster array 1112 can be configuredto perform different types of processing. In at least one embodiment, inat least one embodiment, a first portion may be configured to performvertex shading and topology generation, a second portion may beconfigured to perform tessellation and geometry shading, and a thirdportion may be configured to perform pixel shading or other screen spaceoperations, to produce a rendered image for display if a simulation ofvalve control for a multiple mode cooling subsystem having arepurposable refrigerant cooling subsystem is required. In at least oneembodiment, intermediate data produced by one or more of clusters1114A-1114N may be stored in buffers to allow intermediate data to betransmitted between clusters 1114A-1114N for further processing.

In at least one embodiment, processing cluster array 1112 can receiveprocessing tasks to be executed via scheduler 1110, which receivescommands defining processing tasks from front end 1108. In at least oneembodiment, processing tasks can include indices of data to beprocessed, such as surface (patch) data, primitive data, vertex data,and/or pixel data, as well as state parameters and commands defining howdata is to be processed (such as what program is to be executed). In atleast one embodiment, scheduler 1110 may be configured to fetch indicescorresponding to tasks or may receive indices from front end 1108. In atleast one embodiment, front end 1108 can be configured to ensureprocessing cluster array 1112 is configured to a valid state before aworkload specified by incoming command buffers (such as batch-buffers,push buffers, etc.) is initiated.

In at least one embodiment, each of one or more instances of parallelprocessing unit 1102 can couple with parallel processor memory 1122. Inat least one embodiment, parallel processor memory 1122 can be accessedvia memory crossbar 1116, which can receive memory requests fromprocessing cluster array 1112 as well as I/O unit 1104. In at least oneembodiment, memory crossbar 1116 can access parallel processor memory1122 via a memory interface 1118. In at least one embodiment, memoryinterface 1118 can include multiple partition units (such as partitionunit 1120A, partition unit 1120B, through partition unit 1120N) that caneach couple to a portion (such as memory unit) of parallel processormemory 1122. In at least one embodiment, a number of partition units1120A-1120N is configured to be equal to a number of memory units, suchthat a first partition unit 1120A has a corresponding first memory unit1124A, a second partition unit 1120B has a corresponding memory unit1124B, and a Nth partition unit 1120N has a corresponding Nth memoryunit 1124N. In at least one embodiment, a number of partition units1120A-1120N may not be equal to a number of memory devices.

In at least one embodiment, memory units 1124A-1124N can include varioustypes of memory devices, including dynamic random access memory (DRAM)or graphics random access memory, such as synchronous graphics randomaccess memory (SGRAM), including graphics double data rate (GDDR)memory. In at least one embodiment, memory units 1124A-1124N may alsoinclude 3D stacked memory, including but not limited to high bandwidthmemory (HBM). In at least one embodiment, render targets, such as framebuffers or texture maps may be stored across memory units 1124A-1124N,allowing partition units 1120A-1120N to write portions of each rendertarget in parallel to efficiently use available bandwidth of parallelprocessor memory 1122. In at least one embodiment, a local instance ofparallel processor memory 1122 may be excluded in favor of a unifiedmemory design that utilizes system memory in conjunction with localcache memory.

In at least one embodiment, any one of clusters 1114A-1114N ofprocessing cluster array 1112 can process data that will be written toany of memory units 1124A-1124N within parallel processor memory 1122.In at least one embodiment, memory crossbar 1116 can be configured totransfer an output of each cluster 1114A-1114N to any partition unit1120A-1120N or to another cluster 1114A-1114N, which can performadditional processing operations on an output. In at least oneembodiment, each cluster 1114A-1114N can communicate with memoryinterface 1118 through memory crossbar 1116 to read from or write tovarious external memory devices. In at least one embodiment, memorycrossbar 1116 has a connection to memory interface 1118 to communicatewith I/O unit 1104, as well as a connection to a local instance ofparallel processor memory 1122, enabling processing units withindifferent processing clusters 1114A-1114N to communicate with systemmemory or other memory that is not local to parallel processing unit1102. In at least one embodiment, memory crossbar 1116 can use virtualchannels to separate traffic streams between clusters 1114A-1114N andpartition units 1120A-1120N.

In at least one embodiment, multiple instances of parallel processingunit 1102 can be provided on a single add-in card, or multiple add-incards can be interconnected. In at least one embodiment, differentinstances of parallel processing unit 1102 can be configured tointer-operate even if different instances have different numbers ofprocessing cores, different amounts of local parallel processor memory,and/or other configuration differences. In at least one embodiment, inat least one embodiment, some instances of parallel processing unit 1102can include higher precision floating point units relative to otherinstances. In at least one embodiment, systems incorporating one or moreinstances of parallel processing unit 1102 or parallel processor 1100Bcan be implemented in a variety of configurations and form factors,including but not limited to desktop, laptop, or handheld personalcomputers, servers, workstations, game consoles, and/or embeddedsystems.

FIG. 11C is a block diagram of a partition unit 1120 according to atleast one embodiment. In at least one embodiment, partition unit 1120 isan instance of one of partition units 1120A-1120N of FIG. 11B. In atleast one embodiment, partition unit 1120 includes an L2 cache 1121, aframe buffer interface 1125, and a raster operations unit (“ROP”) 1126.L2 cache 1121 is a read/write cache that is configured to perform loadand store operations received from memory crossbar 1116 and ROP 1126. Inat least one embodiment, read misses and urgent write-back requests areoutput by L2 cache 1121 to frame buffer interface 1125 for processing.In at least one embodiment, updates can also be sent to a frame buffervia frame buffer interface 1125 for processing. In at least oneembodiment, frame buffer interface 1125 interfaces with one of memoryunits in parallel processor memory, such as memory units 1124A-1124N ofFIG. 11B (such as within parallel processor memory 1122).

In at least one embodiment, ROP 1126 is a processing unit that performsraster operations such as stencil, z test, blending, and so forth. In atleast one embodiment, ROP 1126 then outputs processed graphics data thatis stored in graphics memory. In at least one embodiment, ROP 1126includes compression logic to compress depth or color data that iswritten to memory and decompress depth or color data that is read frommemory. In at least one embodiment, compression logic can be losslesscompression logic that makes use of one or more of multiple compressionalgorithms. Compression logic that is performed by ROP 1126 can varybased on statistical characteristics of data to be compressed. In atleast one embodiment, in at least one embodiment, delta colorcompression is performed on depth and color data on a per-tile basis.

In at least one embodiment, ROP 1126 is included within each processingcluster (such as cluster 1114A-1114N of FIG. 11B) instead of withinpartition unit 1120. In at least one embodiment, read and write requestsfor pixel data are transmitted over memory crossbar 1116 instead ofpixel fragment data. In at least one embodiment, processed graphics datamay be displayed on a display device, such as one of one or more displaydevice(s) 1110 of FIG. 11, routed for further processing by processor(s)1102, or routed for further processing by one of processing entitieswithin parallel processor 1100B of FIG. 11B.

FIG. 11D is a block diagram of a processing cluster 1114 within aparallel processing unit according to at least one embodiment. In atleast one embodiment, a processing cluster is an instance of one ofprocessing clusters 1114A-1114N of FIG. 11B. In at least one embodiment,one of more of processing cluster(s) 1114 can be configured to executemany threads in parallel, where “thread” refers to an instance of aparticular program executing on a particular set of input data. In atleast one embodiment, single-instruction, multiple-data (SIMD)instruction issue techniques are used to support parallel execution of alarge number of threads without providing multiple independentinstruction units. In at least one embodiment, single-instruction,multiple-thread (SIMT) techniques are used to support parallel executionof a large number of synchronized threads, using a common instructionunit configured to issue instructions to a set of processing engineswithin each one of processing clusters.

In at least one embodiment, operation of processing cluster 1114 can becontrolled via a pipeline manager 1132 that distributes processing tasksto SIMT parallel processors. In at least one embodiment, pipelinemanager 1132 receives instructions from scheduler 1110 of FIG. 11B andmanages execution of those instructions via a graphics multiprocessor1134 and/or a texture unit 1136. In at least one embodiment, graphicsmultiprocessor 1134 is an exemplary instance of a SIMT parallelprocessor. However, in at least one embodiment, various types of SIMTparallel processors of differing architectures may be included withinprocessing cluster 1114. In at least one embodiment, one or moreinstances of graphics multiprocessor 1134 can be included within aprocessing cluster 1114. In at least one embodiment, graphicsmultiprocessor 1134 can process data and a data crossbar 1140 can beused to distribute processed data to one of multiple possibledestinations, including other shader units. In at least one embodiment,pipeline manager 1132 can facilitate distribution of processed data byspecifying destinations for processed data to be distributed vis datacrossbar 1140.

In at least one embodiment, each graphics multiprocessor 1134 withinprocessing cluster 1114 can include an identical set of functionalexecution logic (such as arithmetic logic units, load-store units,etc.). In at least one embodiment, functional execution logic can beconfigured in a pipelined manner in which new instructions can be issuedbefore previous instructions are complete. In at least one embodiment,functional execution logic supports a variety of operations includinginteger and floating point arithmetic, comparison operations, Booleanoperations, bit-shifting, and computation of various algebraicfunctions. In at least one embodiment, same functional-unit hardware canbe leveraged to perform different operations and any combination offunctional units may be present.

In at least one embodiment, instructions transmitted to processingcluster 1114 constitute a thread. In at least one embodiment, a set ofthreads executing across a set of parallel processing engines is athread group. In at least one embodiment, thread group executes aprogram on different input data. In at least one embodiment, each threadwithin a thread group can be assigned to a different processing enginewithin a graphics multiprocessor 1134. In at least one embodiment, athread group may include fewer threads than a number of processingengines within graphics multiprocessor 1134. In at least one embodiment,when a thread group includes fewer threads than a number of processingengines, one or more processing engines may be idle during cycles inwhich that thread group is being processed. In at least one embodiment,a thread group may also include more threads than a number of processingengines within graphics multiprocessor 1134. In at least one embodiment,when a thread group includes more threads than processing engines withingraphics multiprocessor 1134, processing can be performed overconsecutive clock cycles. In at least one embodiment, multiple threadgroups can be executed concurrently on a graphics multiprocessor 1134.

In at least one embodiment, graphics multiprocessor 1134 includes aninternal cache memory to perform load and store operations. In at leastone embodiment, graphics multiprocessor 1134 can forego an internalcache and use a cache memory (such as L1 cache 1148) within processingcluster 1114. In at least one embodiment, each graphics multiprocessor1134 also has access to L2 caches within partition units (such aspartition units 1120A-1120N of FIG. 11B) that are shared among allprocessing clusters 1114 and may be used to transfer data betweenthreads. In at least one embodiment, graphics multiprocessor 1134 mayalso access off-chip global memory, which can include one or more oflocal parallel processor memory and/or system memory. In at least oneembodiment, any memory external to parallel processing unit 1102 may beused as global memory. In at least one embodiment, processing cluster1114 includes multiple instances of graphics multiprocessor 1134 canshare common instructions and data, which may be stored in L1 cache1148.

In at least one embodiment, each processing cluster 1114 may include amemory management unit (“MMU”) 1145 that is configured to map virtualaddresses into physical addresses. In at least one embodiment, one ormore instances of MMU 1145 may reside within memory interface 1118 ofFIG. 11B. In at least one embodiment, MMU 1145 includes a set of pagetable entries (PTEs) used to map a virtual address to a physical addressof a tile and, in at least one embodiment, a cache line index. In atleast one embodiment, MMU 1145 may include address translation lookasidebuffers (TLB) or caches that may reside within graphics multiprocessor1134 or L1 cache or processing cluster 1114. In at least one embodiment,physical address is processed to distribute surface data access localityto allow efficient request interleaving among partition units. In atleast one embodiment, cache line index may be used to determine whethera request for a cache line is a hit or miss.

In at least one embodiment, a processing cluster 1114 may be configuredsuch that each graphics multiprocessor 1134 is coupled to a texture unit1136 for performing texture mapping operations, such as determiningtexture sample positions, reading texture data, and filtering texturedata. In at least one embodiment, texture data is read from an internaltexture L1 cache (not shown) or from an L1 cache within graphicsmultiprocessor 1134 and is fetched from an L2 cache, local parallelprocessor memory, or system memory, as needed. In at least oneembodiment, each graphics multiprocessor 1134 outputs processed tasks todata crossbar 1140 to provide processed task(s) to another processingcluster 1114 for further processing or to store processed task(s) in anL2 cache, local parallel processor memory, or system memory via memorycrossbar 1116. In at least one embodiment, preROP 1142 (pre-rasteroperations unit) is configured to receive data from graphicsmultiprocessor 1134, direct data to ROP units, which may be located withpartition units as described herein (such as partition units 1120A-1120Nof FIG. 11B). In at least one embodiment, PreROP 1142 unit can performoptimizations for color blending, organize pixel color data, and performaddress translations.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in graphics processingcluster 1114 for inferencing or predicting operations based, at least inpart, on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIG. 11E shows a graphics multiprocessor 1134 according to at least oneembodiment. In at least one embodiment, graphics multiprocessor 1134couples with pipeline manager 1132 of processing cluster 1114. In atleast one embodiment, graphics multiprocessor 1134 has an executionpipeline including but not limited to an instruction cache 1152, aninstruction unit 1154, an address mapping unit 1156, a register file1158, one or more general purpose graphics processing unit (GPGPU) cores1162, and one or more load/store units 1166. GPGPU core(s) 1162 andload/store unit(s) 1166 are coupled with cache memory 1172 and sharedmemory 1170 via a memory and cache interconnect 1168.

In at least one embodiment, instruction cache 1152 receives a stream ofinstructions to execute from pipeline manager 1132. In at least oneembodiment, instructions are cached in instruction cache 1152 anddispatched for execution by instruction unit 1154. In at least oneembodiment, instruction unit 1154 can dispatch instructions as threadgroups (such as warps), with each thread group assigned to a differentexecution unit within GPGPU core(s) 1162. In at least one embodiment, aninstruction can access any of a local, shared, or global address spaceby specifying an address within a unified address space. In at least oneembodiment, address mapping unit 1156 can be used to translate addressesin a unified address space into a distinct memory address that can beaccessed by load/store unit(s) 1166.

In at least one embodiment, register file 1158 provides a set ofregisters for functional units of graphics multiprocessor 1134. In atleast one embodiment, register file 1158 provides temporary storage foroperands connected to data paths of functional units (such as GPGPUcores 1162, load/store units 1166) of graphics multiprocessor 1134. Inat least one embodiment, register file 1158 is divided between each offunctional units such that each functional unit is allocated a dedicatedportion of register file 1158. In at least one embodiment, register file1158 is divided between different warps being executed by graphicsmultiprocessor 1134.

In at least one embodiment, GPGPU cores 1162 can each include floatingpoint units (FPUs) and/or integer arithmetic logic units (ALUs) that areused to execute instructions of graphics multiprocessor 1134. GPGPUcores 1162 can be similar in architecture or can differ in architecture.In at least one embodiment, a first portion of GPGPU cores 1162 includea single precision FPU and an integer ALU while a second portion ofGPGPU cores include a double precision FPU. In at least one embodiment,FPUs can implement IEEE 754-2008 standard for floating point arithmeticor enable variable precision floating point arithmetic. In at least oneembodiment, graphics multiprocessor 1134 can additionally include one ormore fixed function or special function units to perform specificfunctions such as copy rectangle or pixel blending operations. In atleast one embodiment one or more of GPGPU cores can also include fixedor special function logic.

In at least one embodiment, GPGPU cores 1162 include SIMD logic capableof performing a single instruction on multiple sets of data. In at leastone embodiment GPGPU cores 1162 can physically execute SIMD4, SIMD8, andSIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32instructions. In at least one embodiment, SIMD instructions for GPGPUcores can be generated at compile time by a shader compiler orautomatically generated when executing programs written and compiled forsingle program multiple data (SPMD) or SIMT architectures. In at leastone embodiment, multiple threads of a program configured for an SIMTexecution model can executed via a single SIMD instruction. In at leastone embodiment, in at least one embodiment, eight SIMT threads thatperform same or similar operations can be executed in parallel via asingle SIMD8 logic unit.

In at least one embodiment, memory and cache interconnect 1168 is aninterconnect network that connects each functional unit of graphicsmultiprocessor 1134 to register file 1158 and to shared memory 1170. Inat least one embodiment, memory and cache interconnect 1168 is acrossbar interconnect that allows load/store unit 1166 to implement loadand store operations between shared memory 1170 and register file 1158.In at least one embodiment, register file 1158 can operate at a samefrequency as GPGPU cores 1162, thus data transfer between GPGPU cores1162 and register file 1158 is very low latency. In at least oneembodiment, shared memory 1170 can be used to enable communicationbetween threads that execute on functional units within graphicsmultiprocessor 1134. In at least one embodiment, cache memory 1172 canbe used as a data cache for example, to cache texture data communicatedbetween functional units and texture unit 1136. In at least oneembodiment, shared memory 1170 can also be used as a program managedcache. In at least one embodiment, threads executing on GPGPU cores 1162can programmatically store data within shared memory in addition toautomatically cached data that is stored within cache memory 1172.

In at least one embodiment, a parallel processor or GPGPU as describedherein is communicatively coupled to host/processor cores to accelerategraphics operations, machine-learning operations, pattern analysisoperations, and various general purpose GPU (GPGPU) functions. In atleast one embodiment, GPU may be communicatively coupled to hostprocessor/cores over a bus or other interconnect (such as a high speedinterconnect such as PCIe or NVLink). In at least one embodiment, GPUmay be integrated on same package or chip as cores and communicativelycoupled to cores over an internal processor bus/interconnect (in atleast one embodiment, internal to package or chip). In at least oneembodiment, regardless of manner in which GPU is connected, processorcores may allocate work to GPU in form of sequences ofcommands/instructions contained in a work descriptor. In at least oneembodiment, GPU then uses dedicated circuitry/logic for efficientlyprocessing these commands/instructions.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in graphicsmultiprocessor 1134 for inferencing or predicting operations based, atleast in part, on weight parameters calculated using neural networktraining operations, neural network functions and/or architectures, orneural network use cases described herein.

FIG. 12A illustrates a multi-GPU computing system 1200A, according to atleast one embodiment. In at least one embodiment, multi-GPU computingsystem 1200A can include a processor 1202 coupled to multiple generalpurpose graphics processing units (GPGPUs) 1206A-D via a host interfaceswitch 1204. In at least one embodiment, host interface switch 1204 is aPCI express switch device that couples processor 1202 to a PCI expressbus over which processor 1202 can communicate with GPGPUs 1206A-D.GPGPUs 1206A-D can interconnect via a set of high-speed point to pointGPU to GPU links 1216. In at least one embodiment, GPU to GPU links 1216connect to each of GPGPUs 1206A-D via a dedicated GPU link. In at leastone embodiment, P2P GPU links 1216 enable direct communication betweeneach of GPGPUs 1206A-D without requiring communication over hostinterface bus 1204 to which processor 1202 is connected. In at least oneembodiment, with GPU-to-GPU traffic directed to P2P GPU links 1216, hostinterface bus 1204 remains available for system memory access or tocommunicate with other instances of multi-GPU computing system 1200A,for example, via one or more network devices. While in at least oneembodiment GPGPUs 1206A-D connect to processor 1202 via host interfaceswitch 1204, in at least one embodiment processor 1202 includes directsupport for P2P GPU links 1216 and can connect directly to GPGPUs1206A-D.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in multi-GPU computingsystem 1200A for inferencing or predicting operations based, at least inpart, on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIG. 12B is a block diagram of a graphics processor 1200B, according toat least one embodiment. In at least one embodiment, graphics processor1200B includes a ring interconnect 1202, a pipeline front-end 1204, amedia engine 1237, and graphics cores 1280A-1280N. In at least oneembodiment, ring interconnect 1202 couples graphics processor 1200B toother processing units, including other graphics processors or one ormore general-purpose processor cores. In at least one embodiment,graphics processor 1200B is one of many processors integrated within amulti-core processing system.

In at least one embodiment, graphics processor 1200B receives batches ofcommands via ring interconnect 1202. In at least one embodiment,incoming commands are interpreted by a command streamer 1203 in pipelinefront-end 1204. In at least one embodiment, graphics processor 1200Bincludes scalable execution logic to perform 3D geometry processing andmedia processing via graphics core(s) 1280A-1280N. In at least oneembodiment, for 3D geometry processing commands, command streamer 1203supplies commands to geometry pipeline 1236. In at least one embodiment,for at least some media processing commands, command streamer 1203supplies commands to a video front end 1234, which couples with a mediaengine 1237. In at least one embodiment, media engine 1237 includes aVideo Quality Engine (VQE) 1230 for video and image post-processing anda multi-format encode/decode (MFX) 1233 engine to providehardware-accelerated media data encode and decode. In at least oneembodiment, geometry pipeline 1236 and media engine 1237 each generateexecution threads for thread execution resources provided by at leastone graphics core 1280A.

In at least one embodiment, graphics processor 1200B includes scalablethread execution resources featuring modular cores 1280A-1280N(sometimes referred to as core slices), each having multiple sub-cores1250A-1250N, 1260A-1260N (sometimes referred to as core sub-slices). Inat least one embodiment, graphics processor 1200B can have any number ofgraphics cores 1280A through 1280N. In at least one embodiment, graphicsprocessor 1200B includes a graphics core 1280A having at least a firstsub-core 1250A and a second sub-core 1260A. In at least one embodiment,graphics processor 1200B is a low power processor with a single sub-core(such as 1250A). In at least one embodiment, graphics processor 1200Bincludes multiple graphics cores 1280A-1280N, each including a set offirst sub-cores 1250A-1250N and a set of second sub-cores 1260A-1260N.In at least one embodiment, each sub-core in first sub-cores 1250A-1250Nincludes at least a first set of execution units 1252A-1252N andmedia/texture samplers 1254A-1254N. In at least one embodiment, eachsub-core in second sub-cores 1260A-1260N includes at least a second setof execution units 1262A-1262N and samplers 1264A-1264N. In at least oneembodiment, each sub-core 1250A-1250N, 1260A-1260N shares a set ofshared resources 1270A-1270N. In at least one embodiment, sharedresources include shared cache memory and pixel operation logic.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,inference and/or training logic 615 may be used in graphics processor1200B for inferencing or predicting operations based, at least in part,on weight parameters calculated using neural network trainingoperations, neural network functions and/or architectures, or neuralnetwork use cases described herein.

FIG. 13 is a block diagram illustrating micro-architecture for aprocessor 1300 that may include logic circuits to perform instructions,according to at least one embodiment. In at least one embodiment,processor 1300 may perform instructions, including x86 instructions, ARMinstructions, specialized instructions for application-specificintegrated circuits (ASICs), etc. In at least one embodiment, processor1300 may include registers to store packed data, such as 64-bit wideMMX™ registers in microprocessors enabled with MMX technology from IntelCorporation of Santa Clara, Calif. In at least one embodiment, MMXregisters, available in both integer and floating point forms, mayoperate with packed data elements that accompany single instruction,multiple data (“SIMD”) and streaming SIMD extensions (“SSE”)instructions. In at least one embodiment, 128-bit wide XMM registersrelating to SSE2, SSE3, SSE4, AVX, or beyond (referred to generically as“SSEx”) technology may hold such packed data operands. In at least oneembodiment, processor 1300 may perform instructions to acceleratemachine learning or deep learning algorithms, training, or inferencing.

In at least one embodiment, processor 1300 includes an in-order frontend (“front end”) 1301 to fetch instructions to be executed and prepareinstructions to be used later in processor pipeline. In at least oneembodiment, front end 1301 may include several units. In at least oneembodiment, an instruction prefetcher 1326 fetches instructions frommemory and feeds instructions to an instruction decoder 1328 which inturn decodes or interprets instructions. In at least one embodiment, inat least one embodiment, instruction decoder 1328 decodes a receivedinstruction into one or more operations called “micro-instructions” or“micro-operations” (also called “micro ops” or “uops”) that machine mayexecute. In at least one embodiment, instruction decoder 1328 parsesinstruction into an opcode and corresponding data and control fieldsthat may be used by micro-architecture to perform operations inaccordance with at least one embodiment. In at least one embodiment, atrace cache 1330 may assemble decoded uops into program orderedsequences or traces in a uop queue 1334 for execution. In at least oneembodiment, when trace cache 1330 encounters a complex instruction, amicrocode ROM 1332 provides uops needed to complete operation.

In at least one embodiment, some instructions may be converted into asingle micro-op, whereas others need several micro-ops to complete fulloperation. In at least one embodiment, if more than four micro-ops areneeded to complete an instruction, instruction decoder 1328 may accessmicrocode ROM 1332 to perform instruction. In at least one embodiment,an instruction may be decoded into a small number of micro-ops forprocessing at instruction decoder 1328. In at least one embodiment, aninstruction may be stored within microcode ROM 1332 should a number ofmicro-ops be needed to accomplish operation. In at least one embodiment,trace cache 1330 refers to an entry point programmable logic array(“PLA”) to determine a correct micro-instruction pointer for readingmicrocode sequences to complete one or more instructions from microcodeROM 1332 in accordance with at least one embodiment. In at least oneembodiment, after microcode ROM 1332 finishes sequencing micro-ops foran instruction, front end 1301 of machine may resume fetching micro-opsfrom trace cache 1330.

In at least one embodiment, out-of-order execution engine (“out of orderengine”) 1303 may prepare instructions for execution. In at least oneembodiment, out-of-order execution logic has a number of buffers tosmooth out and re-order flow of instructions to optimize performance asthey go down pipeline and get scheduled for execution. In at least oneembodiment, out-of-order execution engine 1303 includes, withoutlimitation, an allocator/register renamer 1340, a memory uop queue 1342,an integer/floating point uop queue 1344, a memory scheduler 1346, afast scheduler 1302, a slow/general floating point scheduler(“slow/general FP scheduler”) 1304, and a simple floating pointscheduler (“simple FP scheduler”) 1306. In at least one embodiment, fastschedule 1302, slow/general floating point scheduler 1304, and simplefloating point scheduler 1306 are also collectively referred to hereinas “uop schedulers 1302, 1304, 1306.” In at least one embodiment,allocator/register renamer 1340 allocates machine buffers and resourcesthat each uop needs in order to execute. In at least one embodiment,allocator/register renamer 1340 renames logic registers onto entries ina register file. In at least one embodiment, allocator/register renamer1340 also allocates an entry for each uop in one of two uop queues,memory uop queue 1342 for memory operations and integer/floating pointuop queue 1344 for non-memory operations, in front of memory scheduler1346 and uop schedulers 1302, 1304, 1306. In at least one embodiment,uop schedulers 1302, 1304, 1306 determine when a uop is ready to executebased on readiness of their dependent input register operand sources andavailability of execution resources uops need to complete theiroperation. In at least one embodiment, fast scheduler 1302 of at leastone embodiment may schedule on each half of main clock cycle whileslow/general floating point scheduler 1304 and simple floating pointscheduler 1306 may schedule once per main processor clock cycle. In atleast one embodiment, uop schedulers 1302, 1304, 1306 arbitrate fordispatch ports to schedule uops for execution.

In at least one embodiment, execution block 1311 includes, withoutlimitation, an integer register file/bypass network 1308, a floatingpoint register file/bypass network (“FP register file/bypass network”)1310, address generation units (“AGUs”) 1312 and 1314, fast ArithmeticLogic Units (ALUs) (“fast ALUs”) 1316 and 1318, a slow Arithmetic LogicUnit (“slow ALU”) 1320, a floating point ALU (“FP”) 1322, and a floatingpoint move unit (“FP move”) 1324. In at least one embodiment, integerregister file/bypass network 1308 and floating point registerfile/bypass network 1310 are also referred to herein as “register files1308, 1310.” In at least one embodiment, AGUs 1312 and 1314, fast ALUs1316 and 1318, slow ALU 1320, floating point ALU 1322, and floatingpoint move unit 1324 are also referred to herein as “execution units1312, 1314, 1316, 1318, 1320, 1322, and 1324.” In at least oneembodiment, execution block b11 may include, without limitation, anynumber (including zero) and type of register files, bypass networks,address generation units, and execution units, in any combination.

In at least one embodiment, register files 1308, 1310 may be arrangedbetween uop schedulers 1302, 1304, 1306, and execution units 1312, 1314,1316, 1318, 1320, 1322, and 1324. In at least one embodiment, integerregister file/bypass network 1308 performs integer operations. In atleast one embodiment, floating point register file/bypass network 1310performs floating point operations. In at least one embodiment, each ofregister files 1308, 1310 may include, without limitation, a bypassnetwork that may bypass or forward just completed results that have notyet been written into register file to new dependent uops. In at leastone embodiment, register files 1308, 1310 may communicate data with eachother. In at least one embodiment, integer register file/bypass network1308 may include, without limitation, two separate register files, oneregister file for low-order thirty-two bits of data and a secondregister file for high order thirty-two bits of data. In at least oneembodiment, floating point register file/bypass network 1310 mayinclude, without limitation, 128-bit wide entries because floating pointinstructions typically have operands from 64 to 128 bits in width.

In at least one embodiment, execution units 1312, 1314, 1316, 1318,1320, 1322, 1324 may execute instructions. In at least one embodiment,register files 1308, 1310 store integer and floating point data operandvalues that micro-instructions need to execute. In at least oneembodiment, processor 1300 may include, without limitation, any numberand combination of execution units 1312, 1314, 1316, 1318, 1320, 1322,1324. In at least one embodiment, floating point ALU 1322 and floatingpoint move unit 1324, may execute floating point, MMX, SIMD, AVX andSSE, or other operations, including specialized machine learninginstructions. In at least one embodiment, floating point ALU 1322 mayinclude, without limitation, a 64-bit by 64-bit floating point dividerto execute divide, square root, and remainder micro ops. In at least oneembodiment, instructions involving a floating point value may be handledwith floating point hardware. In at least one embodiment, ALU operationsmay be passed to fast ALUs 1316, 1318. In at least one embodiment, fastALUS 1316, 1318 may execute fast operations with an effective latency ofhalf a clock cycle. In at least one embodiment, most complex integeroperations go to slow ALU 1320 as slow ALU 1320 may include, withoutlimitation, integer execution hardware for long-latency type ofoperations, such as a multiplier, shifts, flag logic, and branchprocessing. In at least one embodiment, memory load/store operations maybe executed by AGUS 1312, 1314. In at least one embodiment, fast ALU1316, fast ALU 1318, and slow ALU 1320 may perform integer operations on64-bit data operands. In at least one embodiment, fast ALU 1316, fastALU 1318, and slow ALU 1320 may be implemented to support a variety ofdata bit sizes including sixteen, thirty-two, 128, 256, etc. In at leastone embodiment, floating point ALU 1322 and floating point move unit1324 may be implemented to support a range of operands having bits ofvarious widths. In at least one embodiment, floating point ALU 1322 andfloating point move unit 1324 may operate on 128-bit wide packed dataoperands in conjunction with SIMD and multimedia instructions.

In at least one embodiment, uop schedulers 1302, 1304, 1306, dispatchdependent operations before parent load has finished executing. In atleast one embodiment, as uops may be speculatively scheduled andexecuted in processor 1300, processor 1300 may also include logic tohandle memory misses. In at least one embodiment, if a data load missesin data cache, there may be dependent operations in flight in pipelinethat have left scheduler with temporarily incorrect data. In at leastone embodiment, a replay mechanism tracks and re-executes instructionsthat use incorrect data. In at least one embodiment, dependentoperations might need to be replayed and independent ones may be allowedto complete. In at least one embodiment, schedulers and replay mechanismof at least one embodiment of a processor may also be designed to catchinstruction sequences for text string comparison operations.

In at least one embodiment, registers may refer to on-board processorstorage locations that may be used as part of instructions to identifyoperands. In at least one embodiment, registers may be those that may beusable from outside of processor (from a programmer's perspective). Inat least one embodiment, registers might not be limited to a particulartype of circuit. Rather, in at least one embodiment, a register maystore data, provide data, and perform functions described herein. In atleast one embodiment, registers described herein may be implemented bycircuitry within a processor using any number of different techniques,such as dedicated physical registers, dynamically allocated physicalregisters using register renaming, combinations of dedicated anddynamically allocated physical registers, etc. In at least oneembodiment, integer registers store 32-bit integer data. A register fileof at least one embodiment also contains eight multimedia SIMD registersfor packed data.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodimentportions or all of inference and/or training logic 615 may beincorporated into execution block 1311 and other memory or registersshown or not shown. In at least one embodiment, in at least oneembodiment, training and/or inferencing techniques described herein mayuse one or more of ALUs illustrated in execution block 1311. Moreover,weight parameters may be stored in on-chip or off-chip memory and/orregisters (shown or not shown) that configure ALUs of execution block1311 to perform one or more machine learning algorithms, neural networkarchitectures, use cases, or training techniques described herein.

FIG. 14 illustrates a deep learning application processor 1400,according to at least one embodiment. In at least one embodiment, deeplearning application processor 1400 uses instructions that, if executedby deep learning application processor 1400, cause deep learningapplication processor 1400 to perform some or all of processes andtechniques described throughout this disclosure. In at least oneembodiment, deep learning application processor 1400 is anapplication-specific integrated circuit (ASIC). In at least oneembodiment, application processor 1400 performs matrix multiplyoperations either “hard-wired” into hardware as a result of performingone or more instructions or both. In at least one embodiment, deeplearning application processor 1400 includes, without limitation,processing clusters 1410(1)-1410(12), Inter-Chip Links (“ICLs”)1420(1)-1420(12), Inter-Chip Controllers (“ICCs”) 1430(1)-1430(2),memory controllers (“Mem Ctrlrs”) 1442(1)-1442(4), high bandwidth memoryphysical layer (“HBM PHY”) 1444(1)-1444(4), a management-controllercentral processing unit (“management-controller CPU”) 1450, a SerialPeripheral Interface, Inter-Integrated Circuit, and General PurposeInput/Output block (“SPI, I2C, GPIO”), a peripheral componentinterconnect express controller and direct memory access block (“PCIeController and DMA”) 1470, and a sixteen-lane peripheral componentinterconnect express port (“PCI Express×16”) 1480.

In at least one embodiment, processing clusters 1410 may perform deeplearning operations, including inference or prediction operations basedon weight parameters calculated one or more training techniques,including those described herein. In at least one embodiment, eachprocessing cluster 1410 may include, without limitation, any number andtype of processors. In at least one embodiment, deep learningapplication processor 1400 may include any number and type of processingclusters 1400. In at least one embodiment, Inter-Chip Links 1420 arebi-directional. In at least one embodiment, Inter-Chip Links 1420 andInter-Chip Controllers 1430 enable multiple deep learning applicationprocessors 1400 to exchange information, including activationinformation resulting from performing one or more machine learningalgorithms embodied in one or more neural networks. In at least oneembodiment, deep learning application processor 1400 may include anynumber (including zero) and type of ICLs 1420 and ICCs 1430.

In at least one embodiment, HBM2s 1440 provide a total of 32 Gigabytes(GB) of memory. HBM2 1440(i) is associated with both memory controller1442(i) and HBM PHY 1444(i). In at least one embodiment, any number ofHBM2s 1440 may provide any type and total amount of high bandwidthmemory and may be associated with any number (including zero) and typeof memory controllers 1442 and HBM PHYs 1444. In at least oneembodiment, SPI, I2C, GPIO 1460, PCIe Controller and DMA 1470, and/orPCIe 1480 may be replaced with any number and type of blocks that enableany number and type of communication standards in any technicallyfeasible fashion.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment, deeplearning application processor 1400 is used to train a machine learningmodel, such as a neural network, to predict or infer informationprovided to deep learning application processor 1400. In at least oneembodiment, deep learning application processor 1400 is used to infer orpredict information based on a trained machine learning model (such asneural network) that has been trained by another processor or system orby deep learning application processor 1400. In at least one embodiment,processor 1400 may be used to perform one or more neural network usecases described herein.

FIG. 15 is a block diagram of a neuromorphic processor 1500, accordingto at least one embodiment. In at least one embodiment, neuromorphicprocessor 1500 may receive one or more inputs from sources external toneuromorphic processor 1500. In at least one embodiment, these inputsmay be transmitted to one or more neurons 1502 within neuromorphicprocessor 1500. In at least one embodiment, neurons 1502 and componentsthereof may be implemented using circuitry or logic, including one ormore arithmetic logic units (ALUs). In at least one embodiment,neuromorphic processor 1500 may include, without limitation, thousandsor millions of instances of neurons 1502, but any suitable number ofneurons 1502 may be used. In at least one embodiment, each instance ofneuron 1502 may include a neuron input 1504 and a neuron output 1506. Inat least one embodiment, neurons 1502 may generate outputs that may betransmitted to inputs of other instances of neurons 1502. In at leastone embodiment, in at least one embodiment, neuron inputs 1504 andneuron outputs 1506 may be interconnected via synapses 1508.

In at least one embodiment, neurons 1502 and synapses 1508 may beinterconnected such that neuromorphic processor 1500 operates to processor analyze information received by neuromorphic processor 1500. In atleast one embodiment, neurons 1502 may transmit an output pulse (or“fire” or “spike”) when inputs received through neuron input 1504 exceeda threshold. In at least one embodiment, neurons 1502 may sum orintegrate signals received at neuron inputs 1504. In at least oneembodiment, in at least one embodiment, neurons 1502 may be implementedas leaky integrate-and-fire neurons, wherein if a sum (referred to as a“membrane potential”) exceeds a threshold value, neuron 1502 maygenerate an output (or “fire”) using a transfer function such as asigmoid or threshold function. In at least one embodiment, a leakyintegrate-and-fire neuron may sum signals received at neuron inputs 1504into a membrane potential and may also apply a decay factor (or leak) toreduce a membrane potential. In at least one embodiment, a leakyintegrate-and-fire neuron may fire if multiple input signals arereceived at neuron inputs 1504 rapidly enough to exceed a thresholdvalue (in at least one embodiment, this is before a membrane potentialdecays too low to fire). In at least one embodiment, neurons 1502 may beimplemented using circuits or logic that receive inputs, integrateinputs into a membrane potential, and decay a membrane potential. In atleast one embodiment, inputs may be averaged, or any other suitabletransfer function may be used. Furthermore, in at least one embodiment,neurons 1502 may include, without limitation, comparator circuits orlogic that generate an output spike at neuron output 1506 when result ofapplying a transfer function to neuron input 1504 exceeds a threshold.In at least one embodiment, once neuron 1502 fires, it may disregardpreviously received input information by, for example, resetting amembrane potential to 0 or another suitable default value. In at leastone embodiment, once membrane potential is reset to 0, neuron 1502 mayresume normal operation after a suitable period of time (or refractoryperiod).

In at least one embodiment, neurons 1502 may be interconnected throughsynapses 1508. In at least one embodiment, synapses 1508 may operate totransmit signals from an output of a first neuron 1502 to an input of asecond neuron 1502. In at least one embodiment, neurons 1502 maytransmit information over more than one instance of synapse 1508. In atleast one embodiment, one or more instances of neuron output 1506 may beconnected, via an instance of synapse 1508, to an instance of neuroninput 1504 in same neuron 1502. In at least one embodiment, an instanceof neuron 1502 generating an output to be transmitted over an instanceof synapse 1508 may be referred to as a “pre-synaptic neuron” withrespect to that instance of synapse 1508. In at least one embodiment, aninstance of neuron 1502 receiving an input transmitted over an instanceof synapse 1508 may be referred to as a “post-synaptic neuron” withrespect to that instance of synapse 1508. Because an instance of neuron1502 may receive inputs from one or more instances of synapse 1508, andmay also transmit outputs over one or more instances of synapse 1508, asingle instance of neuron 1502 may therefore be both a “pre-synapticneuron” and “post-synaptic neuron,” with respect to various instances ofsynapses 1508, in at least one embodiment.

In at least one embodiment, neurons 1502 may be organized into one ormore layers. Each instance of neuron 1502 may have one neuron output1506 that may fan out through one or more synapses 1508 to one or moreneuron inputs 1504. In at least one embodiment, neuron outputs 1506 ofneurons 1502 in a first layer 1510 may be connected to neuron inputs1504 of neurons 1502 in a second layer 1512. In at least one embodiment,layer 1510 may be referred to as a “feed-forward layer.” In at least oneembodiment, each instance of neuron 1502 in an instance of first layer1510 may fan out to each instance of neuron 1502 in second layer 1512.In at least one embodiment, first layer 1510 may be referred to as a“fully connected feed-forward layer.” In at least one embodiment, eachinstance of neuron 1502 in an instance of second layer 1512 may fan outto fewer than all instances of neuron 1502 in a third layer 1514. In atleast one embodiment, second layer 1512 may be referred to as a“sparsely connected feed-forward layer.” In at least one embodiment,neurons 1502 in second layer 1512 may fan out to neurons 1502 inmultiple other layers, including to neurons 1502 in (same) second layer1512. In at least one embodiment, second layer 1512 may be referred toas a “recurrent layer.” In at least one embodiment, neuromorphicprocessor 1500 may include, without limitation, any suitable combinationof recurrent layers and feed-forward layers, including, withoutlimitation, both sparsely connected feed-forward layers and fullyconnected feed-forward layers.

In at least one embodiment, neuromorphic processor 1500 may include,without limitation, a reconfigurable interconnect architecture ordedicated hard wired interconnects to connect synapse 1508 to neurons1502. In at least one embodiment, neuromorphic processor 1500 mayinclude, without limitation, circuitry or logic that allows synapses tobe allocated to different neurons 1502 as needed based on neural networktopology and neuron fan-in/out. In at least one embodiment, in at leastone embodiment, synapses 1508 may be connected to neurons 1502 using aninterconnect fabric, such as network-on-chip, or with dedicatedconnections. In at least one embodiment, synapse interconnections andcomponents thereof may be implemented using circuitry or logic.

FIG. 16A is a block diagram of a processing system, according to atleast one embodiment. In at least one embodiment, system 1600A includesone or more processors 1602 and one or more graphics processors 1608,and may be a single processor desktop system, a multiprocessorworkstation system, or a server system having a large number ofprocessors 1602 or processor cores 1607. In at least one embodiment,system 1600A is a processing platform incorporated within asystem-on-a-chip (SoC) integrated circuit for use in mobile, handheld,or embedded devices.

In at least one embodiment, system 1600A can include, or be incorporatedwithin a server-based gaming platform, a game console, including a gameand media console, a mobile gaming console, a handheld game console, oran online game console. In at least one embodiment, system 1600A is amobile phone, smart phone, tablet computing device or mobile Internetdevice. In at least one embodiment, processing system 1600A can alsoinclude, couple with, or be integrated within a wearable device, such asa smart watch wearable device, smart eyewear device, augmented realitydevice, or virtual reality device. In at least one embodiment,processing system 1600A is a television or set top box device having oneor more processors 1602 and a graphical interface generated by one ormore graphics processors 1608.

In at least one embodiment, one or more processors 1602 each include oneor more processor cores 1607 to process instructions which, whenexecuted, perform operations for system and user software. In at leastone embodiment, each of one or more processor cores 1607 is configuredto process a specific instruction set 1609. In at least one embodiment,instruction set 1609 may facilitate Complex Instruction Set Computing(CISC), Reduced Instruction Set Computing (RISC), or computing via aVery Long Instruction Word (VLIW). In at least one embodiment, processorcores 1607 may each process a different instruction set 1609, which mayinclude instructions to facilitate emulation of other instruction sets.In at least one embodiment, processor core 1607 may also include otherprocessing devices, such a Digital Signal Processor (DSP).

In at least one embodiment, processor 1602 includes cache memory 1604.In at least one embodiment, processor 1602 can have a single internalcache or multiple levels of internal cache. In at least one embodiment,cache memory is shared among various components of processor 1602. In atleast one embodiment, processor 1602 also uses an external cache (suchas a Level-3 (L3) cache or Last Level Cache (LLC)) (not shown), whichmay be shared among processor cores 1607 using known cache coherencytechniques. In at least one embodiment, register file 1606 isadditionally included in processor 1602 which may include differenttypes of registers for storing different types of data (such as integerregisters, floating point registers, status registers, and aninstruction pointer register). In at least one embodiment, register file1606 may include general-purpose registers or other registers.

In at least one embodiment, one or more processor(s) 1602 are coupledwith one or more interface bus(es) 1610 to transmit communicationsignals such as address, data, or control signals between processor 1602and other components in system 1600A. In at least one embodiment,interface bus 1610, in one embodiment, can be a processor bus, such as aversion of a Direct Media Interface (DMI) bus. In at least oneembodiment, interface 1610 is not limited to a DMI bus, and may includeone or more Peripheral Component Interconnect buses (such as PCI, PCIExpress), memory busses, or other types of interface busses. In at leastone embodiment processor(s) 1602 include an integrated memory controller1616 and a platform controller hub 1630. In at least one embodiment,memory controller 1616 facilitates communication between a memory deviceand other components of system 1600A, while platform controller hub(PCH) 1630 provides connections to I/O devices via a local I/O bus.

In at least one embodiment, memory device 1620 can be a dynamic randomaccess memory (DRAM) device, a static random access memory (SRAM)device, flash memory device, phase-change memory device, or some othermemory device having suitable performance to serve as process memory. Inat least one embodiment memory device 1620 can operate as system memoryfor system 1600A, to store data 1622 and instructions 1621 for use whenone or more processors 1602 executes an application or process. In atleast one embodiment, memory controller 1616 also couples with an, in atleast one embodiment, external graphics processor 1612, which maycommunicate with one or more graphics processors 1608 in processors 1602to perform graphics and media operations. In at least one embodiment, adisplay device 1611 can connect to processor(s) 1602. In at least oneembodiment display device 1611 can include one or more of an internaldisplay device, as in a mobile electronic device or a laptop device oran external display device attached via a display interface (such asDisplayPort, etc.). In at least one embodiment, display device 1611 caninclude a head mounted display (HMD) such as a stereoscopic displaydevice for use in virtual reality (VR) applications or augmented reality(AR) applications.

In at least one embodiment, platform controller hub 1630 enablesperipherals to connect to memory device 1620 and processor 1602 via ahigh-speed I/O bus. In at least one embodiment, I/O peripherals include,but are not limited to, an audio controller 1646, a network controller1634, a firmware interface 1628, a wireless transceiver 1626, touchsensors 1625, a data storage device 1624 (such as hard disk drive, flashmemory, etc.). In at least one embodiment, data storage device 1624 canconnect via a storage interface (such as SATA) or via a peripheral bus,such as a Peripheral Component Interconnect bus (such as PCI, PCIExpress). In at least one embodiment, touch sensors 1625 can includetouch screen sensors, pressure sensors, or fingerprint sensors. In atleast one embodiment, wireless transceiver 1626 can be a Wi-Fitransceiver, a Bluetooth transceiver, or a mobile network transceiversuch as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at leastone embodiment, firmware interface 1628 enables communication withsystem firmware, and can be, for example, a unified extensible firmwareinterface (UEFI). In at least one embodiment, network controller 1634can enable a network connection to a wired network. In at least oneembodiment, a high-performance network controller (not shown) coupleswith interface bus 1610. In at least one embodiment, audio controller1646 is a multi-channel high definition audio controller. In at leastone embodiment, system 1600A includes a legacy I/O controller 1640 forcoupling legacy (such as Personal System 2 (PS/2)) devices to system. Inat least one embodiment, platform controller hub 1630 can also connectto one or more Universal Serial Bus (USB) controllers 1642 connect inputdevices, such as keyboard and mouse 1643 combinations, a camera 1644, orother USB input devices.

In at least one embodiment, an instance of memory controller 1616 andplatform controller hub 1630 may be integrated into a discreet externalgraphics processor, such as external graphics processor 1612. In atleast one embodiment, platform controller hub 1630 and/or memorycontroller 1616 may be external to one or more processor(s) 1602. In atleast one embodiment, in at least one embodiment, system 1600A caninclude an external memory controller 1616 and platform controller hub1630, which may be configured as a memory controller hub and peripheralcontroller hub within a system chipset that is in communication withprocessor(s) 1602.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodimentportions or all of inference and/or training logic 615 may beincorporated into graphics processor 1600A. In at least one embodiment,in at least one embodiment, training and/or inferencing techniquesdescribed herein may use one or more of ALUs embodied in graphicsprocessor 1612. Moreover, in at least one embodiment, inferencing and/ortraining operations described herein may be done using logic other thanlogic illustrated in FIG. 6B or 6C. In at least one embodiment, weightparameters may be stored in on-chip or off-chip memory and/or registers(shown or not shown) that configure ALUs of graphics processor 1600A toperform one or more machine learning algorithms, neural networkarchitectures, use cases, or training techniques described herein.

FIG. 16B is a block diagram of a processor 1600B having one or moreprocessor cores 1602A-1602N, an integrated memory controller 1614, andan integrated graphics processor 1608, according to at least oneembodiment. In at least one embodiment, processor 1600B can includeadditional cores up to and including additional core 1602N representedby dashed lined boxes. In at least one embodiment, each of processorcores 1602A-1602N includes one or more internal cache units 1604A-1604N.In at least one embodiment, each processor core also has access to oneor more shared cached units 1606.

In at least one embodiment, internal cache units 1604A-1604N and sharedcache units 1606 represent a cache memory hierarchy within processor1600B. In at least one embodiment, cache memory units 1604A-1604N mayinclude at least one level of instruction and data cache within eachprocessor core and one or more levels of shared mid-level cache, such asa Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache,where a highest level of cache before external memory is classified asan LLC. In at least one embodiment, cache coherency logic maintainscoherency between various cache units 1606 and 1604A-1604N.

In at least one embodiment, processor 1600B may also include a set ofone or more bus controller units 1616 and a system agent core 1610. Inat least one embodiment, one or more bus controller units 1616 manage aset of peripheral buses, such as one or more PCI or PCI express busses.In at least one embodiment, system agent core 1610 provides managementfunctionality for various processor components. In at least oneembodiment, system agent core 1610 includes one or more integratedmemory controllers 1614 to manage access to various external memorydevices (not shown).

In at least one embodiment, one or more of processor cores 1602A-1602Ninclude support for simultaneous multi-threading. In at least oneembodiment, system agent core 1610 includes components for coordinatingand operating cores 1602A-1602N during multi-threaded processing. In atleast one embodiment, system agent core 1610 may additionally include apower control unit (PCU), which includes logic and components toregulate one or more power states of processor cores 1602A-1602N andgraphics processor 1608.

In at least one embodiment, processor 1600B additionally includesgraphics processor 1608 to execute graphics processing operations. In atleast one embodiment, graphics processor 1608 couples with shared cacheunits 1606, and system agent core 1610, including one or more integratedmemory controllers 1614. In at least one embodiment, system agent core1610 also includes a display controller 1611 to drive graphics processoroutput to one or more coupled displays. In at least one embodiment,display controller 1611 may also be a separate module coupled withgraphics processor 1608 via at least one interconnect, or may beintegrated within graphics processor 1608.

In at least one embodiment, a ring based interconnect unit 1612 is usedto couple internal components of processor 1600B. In at least oneembodiment, an alternative interconnect unit may be used, such as apoint-to-point interconnect, a switched interconnect, or othertechniques. In at least one embodiment, graphics processor 1608 coupleswith ring interconnect 1612 via an I/O link 1613.

In at least one embodiment, I/O link 1613 represents at least one ofmultiple varieties of I/O interconnects, including an on package I/Ointerconnect which facilitates communication between various processorcomponents and a high-performance embedded memory module 1618, such asan eDRAM module. In at least one embodiment, each of processor cores1602A-1602N and graphics processor 1608 use embedded memory modules 1618as a shared Last Level Cache.

In at least one embodiment, processor cores 1602A-1602N are homogenouscores executing a common instruction set architecture. In at least oneembodiment, processor cores 1602A-1602N are heterogeneous in terms ofinstruction set architecture (ISA), where one or more of processor cores1602A-1602N execute a common instruction set, while one or more othercores of processor cores 1602A-16-02N executes a subset of a commoninstruction set or a different instruction set. In at least oneembodiment, processor cores 1602A-1602N are heterogeneous in terms ofmicroarchitecture, where one or more cores having a relatively higherpower consumption couple with one or more power cores having a lowerpower consumption. In at least one embodiment, processor 1600B can beimplemented on one or more chips or as an SoC integrated circuit.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodimentportions or all of inference and/or training logic 615 may beincorporated into processor 1600B. In at least one embodiment, in atleast one embodiment, training and/or inferencing techniques describedherein may use one or more of ALUs embodied in graphics processor 1612,graphics core(s) 1602A-1602N, or other components in FIG. 16. Moreover,in at least one embodiment, inferencing and/or training operationsdescribed herein may be done using logic other than logic illustrated inFIG. 6B or 6C. In at least one embodiment, weight parameters may bestored in on-chip or off-chip memory and/or registers (shown or notshown) that configure ALUs of graphics processor 1600B to perform one ormore machine learning algorithms, neural network architectures, usecases, or training techniques described herein.

FIG. 16C is a block diagram of hardware logic of a graphics processorcore 1600C, according to at least one embodiment described herein. In atleast one embodiment, graphics processor core 1600C is included within agraphics core array. In at least one embodiment, graphics processor core1600C, sometimes referred to as a core slice, can be one or multiplegraphics cores within a modular graphics processor. In at least oneembodiment, graphics processor core 1600C is exemplary of one graphicscore slice, and a graphics processor as described herein may includemultiple graphics core slices based on target power and performanceenvelopes. In at least one embodiment, each graphics core 1600C caninclude a fixed function block 1630 coupled with multiple sub-cores1601A-1601F, also referred to as sub-slices, that include modular blocksof general-purpose and fixed function logic.

In at least one embodiment, fixed function block 1630 includes ageometry/fixed function pipeline 1636 that can be shared by allsub-cores in graphics processor 1600C, for example, in lower performanceand/or lower power graphics processor implementations. In at least oneembodiment, geometry/fixed function pipeline 1636 includes a 3D fixedfunction pipeline, a video front-end unit, a thread spawner and threaddispatcher, and a unified return buffer manager, which manages unifiedreturn buffers.

In at least one embodiment fixed, function block 1630 also includes agraphics SoC interface 1637, a graphics microcontroller 1638, and amedia pipeline 1639. In at least one embodiment fixed, graphics SoCinterface 1637 provides an interface between graphics core 1600C andother processor cores within a system on a chip integrated circuit. Inat least one embodiment, graphics microcontroller 1638 is a programmablesub-processor that is configurable to manage various functions ofgraphics processor 1600C, including thread dispatch, scheduling, andpre-emption. In at least one embodiment, media pipeline 1639 includeslogic to facilitate decoding, encoding, pre-processing, and/orpost-processing of multimedia data, including image and video data. Inat least one embodiment, media pipeline 1639 implements media operationsvia requests to compute or sampling logic within sub-cores 1601-1601F.

In at least one embodiment, SoC interface 1637 enables graphics core1600C to communicate with general-purpose application processor cores(such as CPUs) and/or other components within an SoC, including memoryhierarchy elements such as a shared last level cache memory, system RAM,and/or embedded on-chip or on-package DRAM. In at least one embodiment,SoC interface 1637 can also enable communication with fixed functiondevices within an SoC, such as camera imaging pipelines, and enables useof and/or implements global memory atomics that may be shared betweengraphics core 1600C and CPUs within an SoC. In at least one embodiment,SoC interface 1637 can also implement power management controls forgraphics core 1600C and enable an interface between a clock domain ofgraphic core 1600C and other clock domains within an SoC. In at leastone embodiment, SoC interface 1637 enables receipt of command buffersfrom a command streamer and global thread dispatcher that are configuredto provide commands and instructions to each of one or more graphicscores within a graphics processor. In at least one embodiment, commandsand instructions can be dispatched to media pipeline 1639, when mediaoperations are to be performed, or a geometry and fixed functionpipeline (such as geometry and fixed function pipeline 1636, geometryand fixed function pipeline 1614) when graphics processing operationsare to be performed.

In at least one embodiment, graphics microcontroller 1638 can beconfigured to perform various scheduling and management tasks forgraphics core 1600C. In at least one embodiment, graphicsmicrocontroller 1638 can perform graphics and/or compute workloadscheduling on various graphics parallel engines within execution unit(EU) arrays 1602A-1602F, 1604A-1604F within sub-cores 1601A-1601F. In atleast one embodiment, host software executing on a CPU core of an SoCincluding graphics core 1600C can submit workloads one of multiplegraphic processor doorbells, which invokes a scheduling operation on anappropriate graphics engine. In at least one embodiment, schedulingoperations include determining which workload to run next, submitting aworkload to a command streamer, pre-empting existing workloads runningon an engine, monitoring progress of a workload, and notifying hostsoftware when a workload is complete. In at least one embodiment,graphics microcontroller 1638 can also facilitate low-power or idlestates for graphics core 1600C, providing graphics core 1600C with anability to save and restore registers within graphics core 1600C acrosslow-power state transitions independently from an operating systemand/or graphics driver software on a system.

In at least one embodiment, graphics core 1600C may have greater than orfewer than illustrated sub-cores 1601A-1601F, up to N modular sub-cores.For each set of N sub-cores, in at least one embodiment, graphics core1600C can also include shared function logic 1610, shared and/or cachememory 1612, a geometry/fixed function pipeline 1614, as well asadditional fixed function logic 1616 to accelerate various graphics andcompute processing operations. In at least one embodiment, sharedfunction logic 1610 can include logic units (such as sampler, math,and/or inter-thread communication logic) that can be shared by each Nsub-cores within graphics core 1600C. In at least one embodiment fixed,shared and/or cache memory 1612 can be a last-level cache for Nsub-cores 1601A-1601F within graphics core 1600C and can also serve asshared memory that is accessible by multiple sub-cores. In at least oneembodiment, geometry/fixed function pipeline 1614 can be includedinstead of geometry/fixed function pipeline 1636 within fixed functionblock 1630 and can include same or similar logic units.

In at least one embodiment, graphics core 1600C includes additionalfixed function logic 1616 that can include various fixed functionacceleration logic for use by graphics core 1600C. In at least oneembodiment, additional fixed function logic 1616 includes an additionalgeometry pipeline for use in position only shading. In position-onlyshading, at least two geometry pipelines exist, whereas in a fullgeometry pipeline within geometry/fixed function pipeline 1616, 1636,and a cull pipeline, which is an additional geometry pipeline which maybe included within additional fixed function logic 1616. In at least oneembodiment, cull pipeline is a trimmed down version of a full geometrypipeline. In at least one embodiment, a full pipeline and a cullpipeline can execute different instances of an application, eachinstance having a separate context. In at least one embodiment, positiononly shading can hide long cull runs of discarded triangles, enablingshading to be completed earlier in some instances. In at least oneembodiment, in at least one embodiment, cull pipeline logic withinadditional fixed function logic 1616 can execute position shaders inparallel with a main application and generates critical results fasterthan a full pipeline, as cull pipeline fetches and shades positionattribute of vertices, without performing rasterization and rendering ofpixels to a frame buffer. In at least one embodiment, cull pipeline canuse generated critical results to compute visibility information for alltriangles without regard to whether those triangles are culled. In atleast one embodiment, full pipeline (which in this instance may bereferred to as a replay pipeline) can consume visibility information toskip culled triangles to shade only visible triangles that are finallypassed to a rasterization phase.

In at least one embodiment, additional fixed function logic 1616 canalso include machine-learning acceleration logic, such as fixed functionmatrix multiplication logic, for implementations including optimizationsfor machine learning training or inferencing.

In at least one embodiment, within each graphics sub-core 1601A-1601Fincludes a set of execution resources that may be used to performgraphics, media, and compute operations in response to requests bygraphics pipeline, media pipeline, or shader programs. In at least oneembodiment, graphics sub-cores 1601A-1601F include multiple EU arrays1602A-1602F, 1604A-1604F, thread dispatch and inter-thread communication(TD/IC) logic 1603A-1603F, a 3D (such as texture) sampler 1605A-1605F, amedia sampler 1606A-1606F, a shader processor 1607A-1607F, and sharedlocal memory (SLM) 1608A-1608F. EU arrays 1602A-1602F, 1604A-1604F eachinclude multiple execution units, which are general-purpose graphicsprocessing units capable of performing floating-point andinteger/fixed-point logic operations in service of a graphics, media, orcompute operation, including graphics, media, or compute shaderprograms. In at least one embodiment, TD/IC logic 1603A-1603F performslocal thread dispatch and thread control operations for execution unitswithin a sub-core and facilitate communication between threads executingon execution units of a sub-core. In at least one embodiment, 3D sampler1605A-1605F can read texture or other 3D graphics related data intomemory. In at least one embodiment, 3D sampler can read texture datadifferently based on a configured sample state and texture formatassociated with a given texture. In at least one embodiment, mediasampler 1606A-1606F can perform similar read operations based on a typeand format associated with media data. In at least one embodiment, eachgraphics sub-core 1601A-1601F can alternately include a unified 3D andmedia sampler. In at least one embodiment, threads executing onexecution units within each of sub-cores 1601A-1601F can make use ofshared local memory 1608A-1608F within each sub-core, to enable threadsexecuting within a thread group to execute using a common pool ofon-chip memory.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,portions or all of inference and/or training logic 615 may beincorporated into graphics processor 1610. In at least one embodiment,in at least one embodiment, training and/or inferencing techniquesdescribed herein may use one or more of ALUs embodied in graphicsprocessor 1612, graphics microcontroller 1638, geometry & fixed functionpipeline 1614 and 1636, or other logic in FIG. 16B. Moreover, in atleast one embodiment, inferencing and/or training operations describedherein may be done using logic other than logic illustrated in FIG. 6Bor 6C. In at least one embodiment, weight parameters may be stored inon-chip or off-chip memory and/or registers (shown or not shown) thatconfigure ALUs of graphics processor 1600C to perform one or moremachine learning algorithms, neural network architectures, use cases, ortraining techniques described herein.

FIGS. 16D-16E illustrate thread execution logic 1600D including an arrayof processing elements of a graphics processor core according to atleast one embodiment. FIG. 16D illustrates at least one embodiment, inwhich thread execution logic 1600D is used. FIG. 16E illustratesexemplary internal details of an execution unit, according to at leastone embodiment.

As illustrated in FIG. 16D, in at least one embodiment, thread executionlogic 1600D includes a shader processor 1602, a thread dispatcher 1604,instruction cache 1606, a scalable execution unit array including aplurality of execution units 1608A-1608N, sampler(s) 1610, a data cache1612, and a data port 1614. In at least one embodiment a scalableexecution unit array can dynamically scale by enabling or disabling oneor more execution units (such as any of execution unit 1608A, 1608B,1608C, 1608D, through 1608N-1 and 1608N) based on computationalrequirements of a workload, for example. In at least one embodiment,scalable execution units are interconnected via an interconnect fabricthat links to each of execution unit. In at least one embodiment, threadexecution logic 1600D includes one or more connections to memory, suchas system memory or cache memory, through one or more of instructioncache 1606, data port 1614, sampler 1610, and execution units1608A-1608N. In at least one embodiment, each execution unit (such as1608A) is a stand-alone programmable general-purpose computational unitthat is capable of executing multiple simultaneous hardware threadswhile processing multiple data elements in parallel for each thread. Inat least one embodiment, array of execution units 1608A-1608N isscalable to include any number individual execution units.

In at least one embodiment, execution units 1608A-1608N are primarilyused to execute shader programs. In at least one embodiment, shaderprocessor 1602 can process various shader programs and dispatchexecution threads associated with shader programs via a threaddispatcher 1604. In at least one embodiment, thread dispatcher 1604includes logic to arbitrate thread initiation requests from graphics andmedia pipelines and instantiate requested threads on one or moreexecution units in execution units 1608A-1608N. In at least oneembodiment, in at least one embodiment, a geometry pipeline can dispatchvertex, tessellation, or geometry shaders to thread execution logic forprocessing. In at least one embodiment, thread dispatcher 1604 can alsoprocess runtime thread spawning requests from executing shader programs.

In at least one embodiment, execution units 1608A-1608N support aninstruction set that includes native support for many standard 3Dgraphics shader instructions, such that shader programs from graphicslibraries (such as Direct 3D and OpenGL) are executed with a minimaltranslation. In at least one embodiment, execution units support vertexand geometry processing (such as vertex programs, geometry programs,vertex shaders), pixel processing (such as pixel shaders, fragmentshaders) and general-purpose processing (such as compute and mediashaders). In at least one embodiment, each of execution units1608A-1608N, which include one or more arithmetic logic units (ALUs), iscapable of multi-issue single instruction multiple data (SIMD) executionand multi-threaded operation enables an efficient execution environmentdespite higher latency memory accesses. In at least one embodiment, eachhardware thread within each execution unit has a dedicatedhigh-bandwidth register file and associated independent thread-state. Inat least one embodiment, execution is multi-issue per clock to pipelinescapable of integer, single and double precision floating pointoperations, SIMD branch capability, logical operations, transcendentaloperations, and other miscellaneous operations. In at least oneembodiment, while waiting for data from memory or one of sharedfunctions, dependency logic within execution units 1608A-1608N causes awaiting thread to sleep until requested data has been returned. In atleast one embodiment, while a waiting thread is sleeping, hardwareresources may be devoted to processing other threads. In at least oneembodiment, in at least one embodiment, during a delay associated with avertex shader operation, an execution unit can perform operations for apixel shader, fragment shader, or another type of shader program,including a different vertex shader.

In at least one embodiment, each execution unit in execution units1608A-1608N operates on arrays of data elements. In at least oneembodiment, a number of data elements is “execution size,” or number ofchannels for an instruction. In at least one embodiment, an executionchannel is a logical unit of execution for data element access, masking,and flow control within instructions. In at least one embodiment, anumber of channels may be independent of a number of physical ArithmeticLogic Units (ALUs) or Floating Point Units (FPUs) for a particulargraphics processor. In at least one embodiment, execution units1608A-1608N support integer and floating-point data types.

In at least one embodiment, an execution unit instruction set includesSIMD instructions. In at least one embodiment, various data elements canbe stored as a packed data type in a register and an execution unit willprocess various elements based on data size of elements. In at least oneembodiment, in at least one embodiment, when operating on a 256-bit widevector, 256 bits of a vector are stored in a register and an executionunit operates on a vector as four separate 64-bit packed data elements(Quad-Word (QW) size data elements), eight separate 32-bit packed dataelements (Double Word (DW) size data elements), sixteen separate 16-bitpacked data elements (Word (W) size data elements), or thirty-twoseparate 8-bit data elements (byte (B) size data elements). However, inat least one embodiment, different vector widths and register sizes arepossible.

In at least one embodiment, one or more execution units can be combinedinto a fused execution unit 1609A-1609N having thread control logic(1607A-1607N) that is common to fused EUs. In at least one embodiment,multiple EUs can be fused into an EU group. In at least one embodiment,each EU in fused EU group can be configured to execute a separate SIMDhardware thread. Number of EUs in a fused EU group can vary according tovarious embodiments. In at least one embodiment, various SIMD widths canbe performed per-EU, including but not limited to SIMD8, SIMD16, andSIMD32. In at least one embodiment, each fused graphics execution unit1609A-1609N includes at least two execution units. In at least oneembodiment, in at least one embodiment, fused execution unit 1609Aincludes a first EU 1608A, second EU 1608B, and thread control logic1607A that is common to first EU 1608A and second EU 1608B. In at leastone embodiment, thread control logic 1607A controls threads executed onfused graphics execution unit 1609A, allowing each EU within fusedexecution units 1609A-1609N to execute using a common instructionpointer register.

In at least one embodiment, one or more internal instruction caches(such as 1606) are included in thread execution logic 1600D to cachethread instructions for execution units. In at least one embodiment, oneor more data caches (such as 1612) are included to cache thread dataduring thread execution. In at least one embodiment, a sampler 1610 isincluded to provide texture sampling for 3D operations and mediasampling for media operations. In at least one embodiment, sampler 1610includes specialized texture or media sampling functionality to processtexture or media data during a sampling process before providing sampleddata to an execution unit.

During execution, in at least one embodiment, graphics and mediapipelines send thread initiation requests to thread execution logic1600D via thread spawning and dispatch logic. In at least oneembodiment, once a group of geometric objects has been processed andrasterized into pixel data, pixel processor logic (such as pixel shaderlogic, fragment shader logic, etc.) within shader processor 1602 isinvoked to further compute output information and cause results to bewritten to output surfaces (such as color buffers, depth buffers,stencil buffers, etc.). In at least one embodiment, a pixel shader orfragment shader calculates values of various vertex attributes that areto be interpolated across a rasterized object. In at least oneembodiment, pixel processor logic within shader processor 1602 thenexecutes an application programming interface (API)-supplied pixel orfragment shader program. In at least one embodiment, to execute a shaderprogram, shader processor 1602 dispatches threads to an execution unit(such as 1608A) via thread dispatcher 1604. In at least one embodiment,shader processor 1602 uses texture sampling logic in sampler 1610 toaccess texture data in texture maps stored in memory. In at least oneembodiment, arithmetic operations on texture data and input geometrydata compute pixel color data for each geometric fragment, or discardsone or more pixels from further processing.

In at least one embodiment, data port 1614 provides a memory accessmechanism for thread execution logic 1600D to output processed data tomemory for further processing on a graphics processor output pipeline.In at least one embodiment, data port 1614 includes or couples to one ormore cache memories (such as data cache 1612) to cache data for memoryaccess via a data port.

As illustrated in FIG. 16E, in at least one embodiment, a graphicsexecution unit 1608 can include an instruction fetch unit 1637, ageneral register file array (GRF) 1624, an architectural register filearray (ARF) 1626, a thread arbiter 1622, a send unit 1630, a branch unit1632, a set of SIMD floating point units (FPUs) 1634, and, in at leastone embodiment, a set of dedicated integer SIMD ALUs 1635. In at leastone embodiment, GRF 1624 and ARF 1626 includes a set of general registerfiles and architecture register files associated with each simultaneoushardware thread that may be active in graphics execution unit 1608. Inat least one embodiment, per thread architectural state is maintained inARF 1626, while data used during thread execution is stored in GRF 1624.In at least one embodiment, execution state of each thread, includinginstruction pointers for each thread, can be held in thread-specificregisters in ARF 1626.

In at least one embodiment, graphics execution unit 1608 has anarchitecture that is a combination of Simultaneous Multi-Threading (SMT)and fine-grained Interleaved Multi-Threading (IMT). In at least oneembodiment, architecture has a modular configuration that can befine-tuned at design time based on a target number of simultaneousthreads and number of registers per execution unit, where execution unitresources are divided across logic used to execute multiple simultaneousthreads.

In at least one embodiment, graphics execution unit 1608 can co-issuemultiple instructions, which may each be different instructions. In atleast one embodiment, thread arbiter 1622 of graphics execution unitthread 1608 can dispatch instructions to one of send unit 1630, branchunit 1642, or SIMD FPU(s) 1634 for execution. In at least oneembodiment, each execution thread can access 128 general-purposeregisters within GRF 1624, where each register can store 32 bytes,accessible as a SIMD 8-element vector of 32-bit data elements. In atleast one embodiment, each execution unit thread has access to 4 Kbyteswithin GRF 1624, although embodiments are not so limited, and greater orfewer register resources may be provided in other embodiments. In atleast one embodiment, up to seven threads can execute simultaneously,although a number of threads per execution unit can also vary accordingto embodiments. In at least one embodiment, in which seven threads mayaccess 4 Kbytes, GRF 1624 can store a total of 28 Kbytes. In at leastone embodiment, flexible addressing modes can permit registers to beaddressed together to build effectively wider registers or to representstrided rectangular block data structures.

In at least one embodiment, memory operations, sampler operations, andother longer-latency system communications are dispatched via “send”instructions that are executed by message passing send unit 1630. In atleast one embodiment, branch instructions are dispatched to a dedicatedbranch unit 1632 to facilitate SIMD divergence and eventual convergence.

In at least one embodiment graphics execution unit 1608 includes one ormore SIMD floating point units (FPU(s)) 1634 to perform floating-pointoperations. In at least one embodiment, FPU(s) 1634 also support integercomputation. In at least one embodiment FPU(s) 1634 can SIMD execute upto M number of 32-bit floating-point (or integer) operations, or SIMDexecute up to 2M 16-bit integer or 16-bit floating-point operations. Inat least one embodiment, at least one of FPU(s) provides extended mathcapability to support high-throughput transcendental math functions anddouble precision 64-bit floating-point. In at least one embodiment, aset of 8-bit integer SIMD ALUs 1635 are also present, and may bespecifically optimized to perform operations associated with machinelearning computations.

In at least one embodiment, arrays of multiple instances of graphicsexecution unit 1608 can be instantiated in a graphics sub-core grouping(such as a sub-slice). In at least one embodiment, execution unit 1608can execute instructions across a plurality of execution channels. In atleast one embodiment, each thread executed on graphics execution unit1608 is executed on a different channel.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment,portions or all of inference and/or training logic 615 may beincorporated into execution logic 1600D. Moreover, in at least oneembodiment, inferencing and/or training operations described herein maybe done using logic other than logic illustrated in FIG. 6B or 6C. In atleast one embodiment, weight parameters may be stored in on-chip oroff-chip memory and/or registers (shown or not shown) that configureALUs of execution logic 1600D to perform one or more machine learningalgorithms, neural network architectures, use cases, or trainingtechniques described herein.

FIG. 17A illustrates a parallel processing unit (“PPU”) 1700A, accordingto at least one embodiment. In at least one embodiment, PPU 1700A isconfigured with machine-readable code that, if executed by PPU 1700A,causes PPU 1700A to perform some or all of processes and techniquesdescribed throughout this disclosure. In at least one embodiment, PPU1700A is a multi-threaded processor that is implemented on one or moreintegrated circuit devices and that utilizes multithreading as alatency-hiding technique designed to process computer-readableinstructions (also referred to as machine-readable instructions orsimply instructions) on multiple threads in parallel. In at least oneembodiment, a thread refers to a thread of execution and is aninstantiation of a set of instructions configured to be executed by PPU1700A. In at least one embodiment, PPU 1700A is a graphics processingunit (“GPU”) configured to implement a graphics rendering pipeline forprocessing three-dimensional (“3D”) graphics data in order to generatetwo-dimensional (“2D”) image data for display on a display device suchas a liquid crystal display (“LCD”) device. In at least one embodiment,PPU 1700A is utilized to perform computations such as linear algebraoperations and machine-learning operations. FIG. 17A illustrates anexample parallel processor for illustrative purposes only and should beconstrued as a non-limiting example of processor architecturescontemplated within scope of this disclosure and that any suitableprocessor may be employed to supplement and/or substitute for same.

In at least one embodiment, one or more PPUs 1700A are configured toaccelerate High Performance Computing (“HPC”), datacenter, and machinelearning applications. In at least one embodiment, PPU 1700A isconfigured to accelerate deep learning systems and applicationsincluding following non-limiting examples: autonomous vehicle platforms,deep learning, high-accuracy speech, image, text recognition systems,intelligent video analytics, molecular simulations, drug discovery,disease diagnosis, weather forecasting, big data analytics, astronomy,molecular dynamics simulation, financial modeling, robotics, factoryautomation, real-time language translation, online search optimizations,and personalized user recommendations, and more.

In at least one embodiment, PPU 1700A includes, without limitation, anInput/Output (“I/O”) unit 1706, a front-end unit 1710, a scheduler unit1712, a work distribution unit 1714, a hub 1716, a crossbar (“Xbar”)1720, one or more general processing clusters (“GPCs”) 1718, and one ormore partition units (“memory partition units”) 1722. In at least oneembodiment, PPU 1700A is connected to a host processor or other PPUs1700A via one or more high-speed GPU interconnects (“GPU interconnects”)1708. In at least one embodiment, PPU 1700A is connected to a hostprocessor or other peripheral devices via an interconnect 1702. In atleast one embodiment, PPU 1700A is connected to a local memorycomprising one or more memory devices (“memory”) 1704. In at least oneembodiment, memory devices 1704 include, without limitation, one or moredynamic random access memory (“DRAM”) devices. In at least oneembodiment, one or more DRAM devices are configured and/or configurableas high-bandwidth memory (“HBM”) subsystems, with multiple DRAM diesstacked within each device.

In at least one embodiment, high-speed GPU interconnect 1708 may referto a wire-based multi-lane communications link that is used by systemsto scale and include one or more PPUs 1700A combined with one or morecentral processing units (“CPUs”), supports cache coherence between PPUs1700A and CPUs, and CPU mastering. In at least one embodiment, dataand/or commands are transmitted by high-speed GPU interconnect 1708through hub 1716 to/from other units of PPU 1700A such as one or morecopy engines, video encoders, video decoders, power management units,and other components which may not be explicitly illustrated in FIG.17A.

In at least one embodiment, I/O unit 1706 is configured to transmit andreceive communications (such as commands, data) from a host processor(not illustrated in FIG. 17A) over system bus 1702. In at least oneembodiment, I/O unit 1706 communicates with host processor directly viasystem bus 1702 or through one or more intermediate devices such as amemory bridge. In at least one embodiment, I/O unit 1706 may communicatewith one or more other processors, such as one or more of PPUs 1700A viasystem bus 1702. In at least one embodiment, I/O unit 1706 implements aPeripheral Component Interconnect Express (“PCIe”) interface forcommunications over a PCIe bus. In at least one embodiment, I/O unit1706 implements interfaces for communicating with external devices.

In at least one embodiment, I/O unit 1706 decodes packets received viasystem bus 1702. In at least one embodiment, at least some packetsrepresent commands configured to cause PPU 1700A to perform variousoperations. In at least one embodiment, I/O unit 1706 transmits decodedcommands to various other units of PPU 1700A as specified by commands.In at least one embodiment, commands are transmitted to front-end unit1710 and/or transmitted to hub 1716 or other units of PPU 1700A such asone or more copy engines, a video encoder, a video decoder, a powermanagement unit, etc. (not explicitly illustrated in FIG. 17A). In atleast one embodiment, I/O unit 1706 is configured to routecommunications between and among various logical units of PPU 1700A.

In at least one embodiment, a program executed by host processor encodesa command stream in a buffer that provides workloads to PPU 1700A forprocessing. In at least one embodiment, a workload comprisesinstructions and data to be processed by those instructions. In at leastone embodiment, buffer is a region in a memory that is accessible (suchas read/write) by both host processor and PPU 1700A—a host interfaceunit may be configured to access buffer in a system memory connected tosystem bus 1702 via memory requests transmitted over system bus 1702 byI/O unit 1706. In at least one embodiment, host processor writes commandstream to buffer and then transmits a pointer to start of command streamto PPU 1700A such that front-end unit 1710 receives pointers to one ormore command streams and manages one or more command streams, readingcommands from command streams and forwarding commands to various unitsof PPU 1700A.

In at least one embodiment, front-end unit 1710 is coupled to schedulerunit 1712 that configures various GPCs 1718 to process tasks defined byone or more command streams. In at least one embodiment, scheduler unit1712 is configured to track state information related to various tasksmanaged by scheduler unit 1712 where state information may indicatewhich of GPCs 1718 a task is assigned to, whether task is active orinactive, a priority level associated with task, and so forth. In atleast one embodiment, scheduler unit 1712 manages execution of aplurality of tasks on one or more of GPCs 1718.

In at least one embodiment, scheduler unit 1712 is coupled to workdistribution unit 1714 that is configured to dispatch tasks forexecution on GPCs 1718. In at least one embodiment, work distributionunit 1714 tracks a number of scheduled tasks received from schedulerunit 1712 and work distribution unit 1714 manages a pending task pooland an active task pool for each of GPCs 1718. In at least oneembodiment, pending task pool comprises a number of slots (such as 32slots) that contain tasks assigned to be processed by a particular GPC1718; active task pool may comprise a number of slots (such as 4 slots)for tasks that are actively being processed by GPCs 1718 such that asone of GPCs 1718 completes execution of a task, that task is evictedfrom active task pool for GPC 1718 and one of other tasks from pendingtask pool is selected and scheduled for execution on GPC 1718. In atleast one embodiment, if an active task is idle on GPC 1718, such aswhile waiting for a data dependency to be resolved, then active task isevicted from GPC 1718 and returned to pending task pool while anothertask in pending task pool is selected and scheduled for execution on GPC1718.

In at least one embodiment, work distribution unit 1714 communicateswith one or more GPCs 1718 via XBar 1720. In at least one embodiment,XBar 1720 is an interconnect network that couples many of units of PPU1700A to other units of PPU 1700A and can be configured to couple workdistribution unit 1714 to a particular GPC 1718. In at least oneembodiment, one or more other units of PPU 1700A may also be connectedto XBar 1720 via hub 1716.

In at least one embodiment, tasks are managed by scheduler unit 1712 anddispatched to one of GPCs 1718 by work distribution unit 1714. GPC 1718is configured to process task and generate results. In at least oneembodiment, results may be consumed by other tasks within GPC 1718,routed to a different GPC 1718 via XBar 1720, or stored in memory 1704.In at least one embodiment, results can be written to memory 1704 viapartition units 1722, which implement a memory interface for reading andwriting data to/from memory 1704. In at least one embodiment, resultscan be transmitted to another PPU 1704 or CPU via high-speed GPUinterconnect 1708. In at least one embodiment, PPU 1700A includes,without limitation, a number U of partition units 1722 that is equal tonumber of separate and distinct memory devices 1704 coupled to PPU1700A. In at least one embodiment, partition unit 1722 will be describedin more detail below in conjunction with FIG. 17C.

In at least one embodiment, a host processor executes a driver kernelthat implements an application programming interface (“API”) thatenables one or more applications executing on host processor to scheduleoperations for execution on PPU 1700A. In at least one embodiment,multiple compute applications are simultaneously executed by PPU 1700Aand PPU 1700A provides isolation, quality of service (“QoS”), andindependent address spaces for multiple compute applications. In atleast one embodiment, an application generates instructions (such as inform of API calls) that cause driver kernel to generate one or moretasks for execution by PPU 1700A and driver kernel outputs tasks to oneor more streams being processed by PPU 1700A. In at least oneembodiment, each task comprises one or more groups of related threads,which may be referred to as a warp. In at least one embodiment, a warpcomprises a plurality of related threads (such as 32 threads) that canbe executed in parallel. In at least one embodiment, cooperating threadscan refer to a plurality of threads including instructions to performtask and that exchange data through shared memory. In at least oneembodiment, threads and cooperating threads are described in moredetail, in accordance with at least one embodiment, in conjunction withFIG. 17C.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment, deeplearning application processor is used to train a machine learningmodel, such as a neural network, to predict or infer informationprovided to PPU 1700A. In at least one embodiment, PPU 1700A is used toinfer or predict information based on a trained machine learning model(such as neural network) that has been trained by another processor orsystem or by PPU 1700A. In at least one embodiment, PPU 1700A may beused to perform one or more neural network use cases described herein.

FIG. 17B illustrates a general processing cluster (“GPC”) 1700B,according to at least one embodiment. In at least one embodiment, GPC1700B is GPC 1718 of FIG. 17A. In at least one embodiment, each GPC1700B includes, without limitation, a number of hardware units forprocessing tasks and each GPC 1700B includes, without limitation, apipeline manager 1702, a pre-raster operations unit (“PROP”) 1704, araster engine 1708, a work distribution crossbar (“WDX”) 1716, a memorymanagement unit (“MMU”) 1718, one or more Data Processing Clusters(“DPCs”) 1706, and any suitable combination of parts.

In at least one embodiment, operation of GPC 1700B is controlled bypipeline manager 1702. In at least one embodiment, pipeline manager 1702manages configuration of one or more DPCs 1706 for processing tasksallocated to GPC 1700B. In at least one embodiment, pipeline manager1702 configures at least one of one or more DPCs 1706 to implement atleast a portion of a graphics rendering pipeline. In at least oneembodiment, DPC 1706 is configured to execute a vertex shader program ona programmable streaming multi-processor (“SM”) 1714. In at least oneembodiment, pipeline manager 1702 is configured to route packetsreceived from a work distribution unit to appropriate logical unitswithin GPC 1700B, in at least one embodiment, and some packets may berouted to fixed function hardware units in PROP 1704 and/or rasterengine 1708 while other packets may be routed to DPCs 1706 forprocessing by a primitive engine 1712 or SM 1714. In at least oneembodiment, pipeline manager 1702 configures at least one of DPCs 1706to implement a neural network model and/or a computing pipeline.

In at least one embodiment, PROP unit 1704 is configured, in at leastone embodiment, to route data generated by raster engine 1708 and DPCs1706 to a Raster Operations (“ROP”) unit in partition unit 1722,described in more detail above in conjunction with FIG. 17A. In at leastone embodiment, PROP unit 1704 is configured to perform optimizationsfor color blending, organize pixel data, perform address translations,and more. In at least one embodiment, raster engine 1708 includes,without limitation, a number of fixed function hardware units configuredto perform various raster operations, in at least one embodiment, andraster engine 1708 includes, without limitation, a setup engine, acoarse raster engine, a culling engine, a clipping engine, a fine rasterengine, a tile coalescing engine, and any suitable combination thereof.In at least one embodiment, setup engine receives transformed verticesand generates plane equations associated with geometric primitivedefined by vertices; plane equations are transmitted to coarse rasterengine to generate coverage information (such as an x, y coverage maskfor a tile) for primitive; output of coarse raster engine is transmittedto culling engine where fragments associated with primitive that fail az-test are culled, and transmitted to a clipping engine where fragmentslying outside a viewing frustum are clipped. In at least one embodiment,fragments that survive clipping and culling are passed to fine rasterengine to generate attributes for pixel fragments based on planeequations generated by setup engine. In at least one embodiment, outputof raster engine 1708 comprises fragments to be processed by anysuitable entity such as by a fragment shader implemented within DPC1706.

In at least one embodiment, each DPC 1706 included in GPC 1700Bcomprise, without limitation, an M-Pipe Controller (“MPC”) 1710;primitive engine 1712; one or more SMs 1714; and any suitablecombination thereof. In at least one embodiment, MPC 1710 controlsoperation of DPC 1706, routing packets received from pipeline manager1702 to appropriate units in DPC 1706. In at least one embodiment,packets associated with a vertex are routed to primitive engine 1712,which is configured to fetch vertex attributes associated with vertexfrom memory; in contrast, packets associated with a shader program maybe transmitted to SM 1714.

In at least one embodiment, SM 1714 comprises, without limitation, aprogrammable streaming processor that is configured to process tasksrepresented by a number of threads. In at least one embodiment, SM 1714is multi-threaded and configured to execute a plurality of threads (suchas 32 threads) from a particular group of threads concurrently andimplements a Single-Instruction, Multiple-Data (“SIMD”) architecturewhere each thread in a group of threads (such as a warp) is configuredto process a different set of data based on same set of instructions. Inat least one embodiment, all threads in group of threads execute sameinstructions. In at least one embodiment, SM 1714 implements aSingle-Instruction, Multiple Thread (“SIMT”) architecture wherein eachthread in a group of threads is configured to process a different set ofdata based on same set of instructions, but where individual threads ingroup of threads are allowed to diverge during execution. In at leastone embodiment, a program counter, call stack, and execution state ismaintained for each warp, enabling concurrency between warps and serialexecution within warps when threads within warp diverge. In anotherembodiment, a program counter, call stack, and execution state ismaintained for each individual thread, enabling equal concurrencybetween all threads, within and between warps. In at least oneembodiment, execution state is maintained for each individual thread andthreads executing same instructions may be converged and executed inparallel for better efficiency. At least one embodiment of SM 1714 aredescribed in more detail below.

In at least one embodiment, MMU 1718 provides an interface between GPC1700B and memory partition unit (such as partition unit 1722 of FIG.17A) and MMU 1718 provides translation of virtual addresses intophysical addresses, memory protection, and arbitration of memoryrequests. In at least one embodiment, MMU 1718 provides one or moretranslation lookaside buffers (“TLB s”) for performing translation ofvirtual addresses into physical addresses in memory.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment, deeplearning application processor is used to train a machine learningmodel, such as a neural network, to predict or infer informationprovided to GPC 1700B. In at least one embodiment, GPC 1700B is used toinfer or predict information based on a trained machine learning model(such as neural network) that has been trained by another processor orsystem or by GPC 1700B. In at least one embodiment, GPC 1700B may beused to perform one or more neural network use cases described herein.

FIG. 17C illustrates a memory partition unit 1700C of a parallelprocessing unit (“PPU”), in accordance with at least one embodiment. Inat least one embodiment, memory partition unit 1700C includes, withoutlimitation, a Raster Operations (“ROP”) unit 1702; a level two (“L2”)cache 1704; a memory interface 1706; and any suitable combinationthereof. In at least one embodiment, memory interface 1706 is coupled tomemory. In at least one embodiment, memory interface 1706 may implement32, 64, 128, 1024-bit data buses, or similar implementations, forhigh-speed data transfer. In at least one embodiment, PPU incorporates Umemory interfaces 1706, one memory interface 1706 per pair of partitionunits 1700C, where each pair of partition units 1700C is connected to acorresponding memory device. In at least one embodiment, in at least oneembodiment, PPU may be connected to up to Y memory devices, such as highbandwidth memory stacks or graphics double-data-rate, version 5,synchronous dynamic random a17ess memory (“GDDR5 SDRAM”).

In at least one embodiment, memory interface 1706 implements a highbandwidth memory second generation (“HBM2”) memory interface and Yequals half U. In at least one embodiment, HBM2 memory stacks arelocated on same physical package as PPU, providing substantial power andarea savings compared with GDDR5 SDRAM systems. In at least oneembodiment, each HBM2 stack includes, without limitation, four memorydies and Y equals 4, with each HBM2 stack including two 128-bit channelsper die for a total of 8 channels and a data bus width of 1024 bits. Inat least one embodiment, memory supports Single-Error CorrectingDouble-Error Detecting (“SECDED”) Error Correction Code (“ECC”) toprotect data. In at least one embodiment, ECC provides higherreliability for compute applications that are sensitive to datacorruption.

In at least one embodiment, PPU implements a multi-level memoryhierarchy. In at least one embodiment, memory partition unit 1700Csupports a unified memory to provide a single unified virtual addressspace for central processing unit (“CPU”) and PPU memory, enabling datasharing between virtual memory systems. In at least one embodiment,frequency of accesses by a PPU to memory located on other processors istraced to ensure that memory pages are moved to physical memory of PPUthat is accessing pages more frequently. In at least one embodiment,high-speed GPU interconnect 1708 supports address translation servicesallowing PPU to directly access a CPU's page tables and providing fullaccess to CPU memory by PPU.

In at least one embodiment, copy engines transfer data between multiplePPUs or between PPUs and CPUs. In at least one embodiment, copy enginescan generate page faults for addresses that are not mapped into pagetables and memory partition unit 1700C then services page faults,mapping addresses into page table, after which copy engine performstransfer. In at least one embodiment, memory is pinned (in at least oneembodiment, non-pageable) for multiple copy engine operations betweenmultiple processors, substantially reducing available memory. In atleast one embodiment, with hardware page faulting, addresses can bepassed to copy engines without regard as to whether memory pages areresident, and copy process is transparent.

Data from memory 1704 of FIG. 17A or other system memory is fetched bymemory partition unit 1700C and stored in L2 cache 1704, which islocated on-chip and is shared between various GPCs, in accordance withat least one embodiment. Each memory partition unit 1700C, in at leastone embodiment, includes, without limitation, at least a portion of L2cache associated with a corresponding memory device. In at least oneembodiment, lower level caches are implemented in various units withinGPCs. In at least one embodiment, each of SMs 1714 may implement a levelone (“L1”) cache wherein L1 cache is private memory that is dedicated toa particular SM 1714 and data from L2 cache 1704 is fetched and storedin each of L1 caches for processing in functional units of SMs 1714. Inat least one embodiment, L2 cache 1704 is coupled to memory interface1706 and XBar 1720.

ROP unit 1702 performs graphics raster operations related to pixelcolor, such as color compression, pixel blending, and more, in at leastone embodiment. ROP unit 1702, in at least one embodiment, implementsdepth testing in conjunction with raster engine 1708, receiving a depthfor a sample location associated with a pixel fragment from cullingengine of raster engine 1708. In at least one embodiment, depth istested against a corresponding depth in a depth buffer for a samplelocation associated with fragment. In at least one embodiment, iffragment passes depth test for sample location, then ROP unit 1702updates depth buffer and transmits a result of depth test to rasterengine 1708. It will be appreciated that number of partition units 1700Cmay be different than number of GPCs and, therefore, each ROP unit 1702can, in at least one embodiment, be coupled to each of GPCs. In at leastone embodiment, ROP unit 1702 tracks packets received from differentGPCs and determines which that a result generated by ROP unit 1702 isrouted to through XBar 1720.

FIG. 17D illustrates a streaming multi-processor (“SM”) 1700D, accordingto at least one embodiment. In at least one embodiment, SM 1700D is SM1714 of FIG. 17B. In at least one embodiment, SM 1700D includes, withoutlimitation, an instruction cache 1702; one or more scheduler units 1704;a register file 1708; one or more processing cores (“cores”) 1710; oneor more special function units (“SFUs”) 1712; one or more load/storeunits (“LSUs”) 1714; an interconnect network 1716; a shared memory/levelone (“L1”) cache 1718; and any suitable combination thereof. In at leastone embodiment, a work distribution unit dispatches tasks for executionon general processing clusters (“GPCs”) of parallel processing units(“PPUs”) and each task is allocated to a particular Data ProcessingCluster (“DPC”) within a GPC and, if task is associated with a shaderprogram, task is allocated to one of SMs 1700D. In at least oneembodiment, scheduler unit 1704 receives tasks from work distributionunit and manages instruction scheduling for one or more thread blocksassigned to SM 1700D. In at least one embodiment, scheduler unit 1704schedules thread blocks for execution as warps of parallel threads,wherein each thread block is allocated at least one warp. In at leastone embodiment, each warp executes threads. In at least one embodiment,scheduler unit 1704 manages a plurality of different thread blocks,allocating warps to different thread blocks and then dispatchinginstructions from plurality of different cooperative groups to variousfunctional units (such as processing cores 1710, SFUs 1712, and LSUs1714) during each clock cycle.

In at least one embodiment, Cooperative Groups may refer to aprogramming model for organizing groups of communicating threads thatallows developers to express granularity at which threads arecommunicating, enabling expression of richer, more efficient paralleldecompositions. In at least one embodiment, cooperative launch APIssupport synchronization amongst thread blocks for execution of parallelalgorithms. In at least one embodiment, applications of programmingmodels provide a single, simple construct for synchronizing cooperatingthreads: a barrier across all threads of a thread block (such assyncthreads( ) function). However, in at least one embodiment,programmers may define groups of threads at smaller than thread blockgranularities and synchronize within defined groups to enable greaterperformance, design flexibility, and software reuse in form ofcollective group-wide function interfaces. In at least one embodiment,Cooperative Groups enables programmers to define groups of threadsexplicitly at sub-block (in at least one embodiment, as small as asingle thread) and multi-block granularities, and to perform collectiveoperations such as synchronization on threads in a cooperative group. Inat least one embodiment, programming model supports clean compositionacross software boundaries, so that libraries and utility functions cansynchronize safely within their local context without having to makeassumptions about convergence. In at least one embodiment, CooperativeGroups primitives enable new patterns of cooperative parallelism,including, without limitation, producer-consumer parallelism,opportunistic parallelism, and global synchronization across an entiregrid of thread blocks.

In at least one embodiment, a dispatch unit 1706 is configured totransmit instructions to one or more of functional units and schedulerunit 1704 includes, without limitation, two dispatch units 1706 thatenable two different instructions from same warp to be dispatched duringeach clock cycle. In at least one embodiment, each scheduler unit 1704includes a single dispatch unit 1706 or additional dispatch units 1706.

In at least one embodiment, each SM 1700D, in at least one embodiment,includes, without limitation, register file 1708 that provides a set ofregisters for functional units of SM 1700D. In at least one embodiment,register file 1708 is divided between each of functional units such thateach functional unit is allocated a dedicated portion of register file1708. In at least one embodiment, register file 1708 is divided betweendifferent warps being executed by SM 1700D and register file 1708provides temporary storage for operands connected to data paths offunctional units. In at least one embodiment, each SM 1700D comprises,without limitation, a plurality of L processing cores 1710. In at leastone embodiment, SM 1700D includes, without limitation, a large number(such as 128 or more) of distinct processing cores 1710. In at least oneembodiment, each processing core 1710, in at least one embodiment,includes, without limitation, a fully-pipelined, single-precision,double-precision, and/or mixed precision processing unit that includes,without limitation, a floating point arithmetic logic unit and aninteger arithmetic logic unit. In at least one embodiment, floatingpoint arithmetic logic units implement IEEE 754-2008 standard forfloating point arithmetic. In at least one embodiment, processing cores1710 include, without limitation, 64 single-precision (32-bit) floatingpoint cores, 64 integer cores, 32 double-precision (64-bit) floatingpoint cores, and 8 tensor cores.

Tensor cores are configured to perform matrix operations in accordancewith at least one embodiment. In at least one embodiment, one or moretensor cores are included in processing cores 1710. In at least oneembodiment, tensor cores are configured to perform deep learning matrixarithmetic, such as convolution operations for neural network trainingand inferencing. In at least one embodiment, each tensor core operateson a 4×4 matrix and performs a matrix multiply and accumulate operationD=A×B+C, where A, B, C, and D are 4×4 matrices.

In at least one embodiment, matrix multiply inputs A and B are 16-bitfloating point matrices and accumulation matrices C and D are 16-bitfloating point or 32-bit floating point matrices. In at least oneembodiment, tensor cores operate on 16-bit floating point input datawith 32-bit floating point accumulation. In at least one embodiment,16-bit floating point multiply uses 64 operations and results in a fullprecision product that is then accumulated using 32-bit floating pointaddition with other intermediate products for a 4×4×4 matrix multiply.Tensor cores are used to perform much larger two-dimensional or higherdimensional matrix operations, built up from these smaller elements, inat least one embodiment. In at least one embodiment, an API, such asCUDA 9 C++ API, exposes specialized matrix load, matrix multiply andaccumulate, and matrix store operations to efficiently use tensor coresfrom a CUDA-C++ program. In at least one embodiment, at CUDA level,warp-level interface assumes 16×16 size matrices spanning all 32 threadsof warp.

In at least one embodiment, each SM 1700D comprises, without limitation,M SFUs 1712 that perform special functions (such as attributeevaluation, reciprocal square root, etc.). In at least one embodiment,SFUs 1712 include, without limitation, a tree traversal unit configuredto traverse a hierarchical tree data structure. In at least oneembodiment, SFUs 1712 include, without limitation, a texture unitconfigured to perform texture map filtering operations. In at least oneembodiment, texture units are configured to load texture maps (such as a2D array of texels) from memory and sample texture maps to producesampled texture values for use in shader programs executed by SM 1700D.In at least one embodiment, texture maps are stored in shared memory/L1cache 1718. In at least one embodiment, texture units implement textureoperations such as filtering operations using mip-maps (such as texturemaps of varying levels of detail), in accordance with at least oneembodiment. In at least one embodiment, each SM 1700D includes, withoutlimitation, two texture units.

Each SM 1700D comprises, without limitation, N LSUs 1714 that implementload and store operations between shared memory/L1 cache 1718 andregister file 1708, in at least one embodiment. Each SM 1700D includes,without limitation, interconnect network 1716 that connects each offunctional units to register file 1708 and LSU 1714 to register file1708 and shared memory/L1 cache 1718 in at least one embodiment. In atleast one embodiment, interconnect network 1716 is a crossbar that canbe configured to connect any of functional units to any of registers inregister file 1708 and connect LSUs 1714 to register file 1708 andmemory locations in shared memory/L1 cache 1718.

In at least one embodiment, shared memory/L1 cache 1718 is an array ofon-chip memory that allows for data storage and communication between SM1700D and primitive engine and between threads in SM 1700D, in at leastone embodiment. In at least one embodiment, shared memory/L1 cache 1718comprises, without limitation, 128 KB of storage capacity and is in pathfrom SM 1700D to partition unit. In at least one embodiment, sharedmemory/L1 cache 1718, in at least one embodiment, is used to cache readsand writes. In at least one embodiment, one or more of shared memory/L1cache 1718, L2 cache, and memory are backing stores.

Combining data cache and shared memory functionality into a singlememory block provides improved performance for both types of memoryaccesses, in at least one embodiment. In at least one embodiment,capacity is used or is usable as a cache by programs that do not useshared memory, such as if shared memory is configured to use half ofcapacity, texture and load/store operations can use remaining capacity.Integration within shared memory/L1 cache 1718 enables shared memory/L1cache 1718 to function as a high-throughput conduit for streaming datawhile simultaneously providing high-bandwidth and low-latency access tofrequently reused data, in accordance with at least one embodiment. Inat least one embodiment, when configured for general purpose parallelcomputation, a simpler configuration can be used compared with graphicsprocessing. In at least one embodiment, fixed function graphicsprocessing units are bypassed, creating a much simpler programmingmodel. In general purpose parallel computation configuration, workdistribution unit assigns and distributes blocks of threads directly toDPCs, in at least one embodiment. In at least one embodiment, threads ina block execute same program, using a unique thread ID in calculation toensure each thread generates unique results, using SM 1700D to executeprogram and perform calculations, shared memory/L1 cache 1718 tocommunicate between threads, and LSU 1714 to read and write globalmemory through shared memory/L1 cache 1718 and memory partition unit. Inat least one embodiment, when configured for general purpose parallelcomputation, SM 1700D writes commands that scheduler unit 1704 can useto launch new work on DPCs.

In at least one embodiment, PPU is included in or coupled to a desktopcomputer, a laptop computer, a tablet computer, servers, supercomputers,a smart-phone (such as a wireless, hand-held device), personal digitalassistant (“PDA”), a digital camera, a vehicle, a head mounted display,a hand-held electronic device, and more. In at least one embodiment, PPUis embodied on a single semiconductor substrate. In at least oneembodiment, PPU is included in a system-on-a-chip (“SoC”) along with oneor more other devices such as additional PPUs, memory, a reducedinstruction set computer (“RISC”) CPU, a memory management unit (“MMU”),a digital-to-analog converter (“DAC”), and like.

In at least one embodiment, PPU may be included on a graphics card thatincludes one or more memory devices. A graphics card may be configuredto interface with a PCIe slot on a motherboard of a desktop computer. Inat least one embodiment, PPU may be an integrated graphics processingunit (“iGPU”) included in chipset of motherboard.

Inference and/or training logic 615 are used to perform inferencingand/or training operations associated with one or more embodiments.Details regarding inference and/or training logic 615 are provided belowin conjunction with FIGS. 6B and/or 6C. In at least one embodiment, deeplearning application processor is used to train a machine learningmodel, such as a neural network, to predict or infer informationprovided to SM 1700D. In at least one embodiment, SM 1700D is used toinfer or predict information based on a trained machine learning model(such as neural network) that has been trained by another processor orsystem or by SM 1700D. In at least one embodiment, SM 1700D may be usedto perform one or more neural network use cases described herein.

In at least one embodiment, a single semiconductor platform may refer toa sole unitary semiconductor-based integrated circuit or chip. In atleast one embodiment, multi-chip modules may be used with increasedconnectivity which simulate on-chip operation, and make substantialimprovements over utilizing a central processing unit (“CPU”) and busimplementation. In at least one embodiment, various modules may also besituated separately or in various combinations of semiconductorplatforms per desires of user.

In at least one embodiment, computer programs in form ofmachine-readable executable code or computer control logic algorithmsare stored in main memory 4 ee 04 and/or secondary storage. Computerprograms, if executed by one or more processors, enable system 4 ee 00to perform various functions in accordance with at least one embodiment.In at least one embodiment, memory 4 ee 04, storage, and/or any otherstorage are possible examples of computer-readable media. In at leastone embodiment, secondary storage may refer to any suitable storagedevice or system such as a hard disk drive and/or a removable storagedrive, representing a floppy disk drive, a magnetic tape drive, acompact disk drive, digital versatile disk (“DVD”) drive, recordingdevice, universal serial bus (“USB”) flash memory, etc. In at least oneembodiment, architecture and/or functionality of various previousfigures are implemented in context of CPU 4 ee 02; parallel processingsystem 4 ee 12; an integrated circuit capable of at least a portion ofcapabilities of both CPU 4 ee 02; parallel processing system 4 ee 12; achipset (such as a group of integrated circuits designed to work andsold as a unit for performing related functions, etc.); and any suitablecombination of integrated circuit(s).

In at least one embodiment, architecture and/or functionality of variousprevious figures are implemented in context of a general computersystem, a circuit board system, a game console system dedicated forentertainment purposes, an application-specific system, and more. In atleast one embodiment, computer system 4 ee 00 may take form of a desktopcomputer, a laptop computer, a tablet computer, servers, supercomputers,a smart-phone (such as a wireless, hand-held device), personal digitalassistant (“PDA”), a digital camera, a vehicle, a head mounted display,a hand-held electronic device, a mobile phone device, a television,workstation, game consoles, embedded system, and/or any other type oflogic.

In at least one embodiment, parallel processing system 4 ee 12 includes,without limitation, a plurality of parallel processing units (“PPUs”) 4ee 14 and associated memories 4 ee 16. In at least one embodiment, PPUs4 ee 14 are connected to a host processor or other peripheral devicesvia an interconnect 4 ee 18 and a switch 4 ee 20 or multiplexer. In atleast one embodiment, parallel processing system 4 ee 12 distributescomputational tasks across PPUs 4 ee 14 which can be parallelizable—forexample, as part of distribution of computational tasks across multiplegraphics processing unit (“GPU”) thread blocks. In at least oneembodiment, memory is shared and accessible (such as for read and/orwrite access) across some or all of PPUs 4 ee 14, although such sharedmemory may incur performance penalties relative to use of local memoryand registers resident to a PPU 4 ee 14. In at least one embodiment,operation of PPUs 4 ee 14 is synchronized through use of a command suchas _syncthreads( ) wherein all threads in a block (such as executedacross multiple PPUs 4 ee 14) to reach a certain point of execution ofcode before proceeding.

Other variations are within spirit of present disclosure. Thus, whiledisclosed techniques are susceptible to various modifications andalternative constructions, certain illustrated embodiments thereof areshown in drawings and have been described above in detail. It should beunderstood, however, that there is no intention to limit disclosure tospecific form or forms disclosed, but on contrary, intention is to coverall modifications, alternative constructions, and equivalents fallingwithin spirit and scope of disclosure, as defined in appended claims.

Use of terms “a” and “an” and “the” and similar referents in context ofdescribing disclosed embodiments (especially in context of followingclaims) are to be construed to cover both singular and plural, unlessotherwise indicated herein or clearly contradicted by context, and notas a definition of a term. Terms “including,” “having,” “including,” and“containing” are to be construed as open-ended terms (meaning“including, but not limited to,”) unless otherwise noted. Term“connected,” when unmodified and referring to physical connections, isto be construed as partly or wholly contained within, attached to, orjoined together, even if there is something intervening. Recitation ofranges of values herein are merely intended to serve as a shorthandmethod of referring individually to each separate value falling withinrange, unless otherwise indicated herein and each separate value isincorporated into specification as if it were individually recitedherein. Use of a set (such as a set of items) or subset, unlessotherwise noted or contradicted by context, is to be construed as anonempty collection including one or more members. Further, unlessotherwise noted or contradicted by context, a subset of a correspondingset does not necessarily denote a proper subset of corresponding set,but subset and corresponding set may be equal.

Conjunctive language, such as phrases of form “at least one of A, B, andC,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of set ofA and B and C. For instance, in illustrative example of a set havingthree members, conjunctive phrases “at least one of A, B, and C” and “atleast one of A, B and C” refer to any of following sets: {A}, {B}, {C},{A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctive language maynot be intended to imply that certain embodiments require at least oneof A, at least one of B, and at least one of C each to be present. Inaddition, unless otherwise noted or contradicted by context, a pluralityindicates a state of being plural (such as a plurality of itemsindicates multiple items). A plurality is at least two items, but can bemore when so indicated either explicitly or by context. Further, unlessstated otherwise or otherwise clear from context, based on means basedat least in part on and not based solely on.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. In at least one embodiment, a process such asthose processes described herein (or variations and/or combinationsthereof) is performed under control of one or more computer systemsconfigured with executable instructions and is implemented as code (suchas executable instructions, one or more computer programs or one or moreapplications) executing collectively on one or more processors, byhardware or combinations thereof. In at least one embodiment, code isstored on a computer-readable storage medium, for example, in form of acomputer program including a plurality of instructions executable by oneor more processors. In at least one embodiment, a computer-readablestorage medium is a non-transitory computer-readable storage medium thatexcludes transitory signals (such as a propagating transient electric orelectromagnetic transmission) but includes non-transitory data storagecircuitry (such as buffers, cache, and queues) within transceivers oftransitory signals. In at least one embodiment, code (such as executablecode or source code) is stored on a set of one or more non-transitorycomputer-readable storage media having stored thereon executableinstructions (or other memory to store executable instructions) that,when executed (in at least one embodiment, as a result of beingexecuted) by one or more processors of a computer system, cause computersystem to perform operations described herein. A set of non-transitorycomputer-readable storage media, in at least one embodiment, includesmultiple non-transitory computer-readable storage media and one or moreof individual non-transitory storage media of multiple non-transitorycomputer-readable storage media lack all of code while multiplenon-transitory computer-readable storage media collectively store all ofcode. In at least one embodiment, executable instructions are executedsuch that different instructions are executed by differentprocessors—for example, a non-transitory computer-readable storagemedium store instructions and a main central processing unit (“CPU”)executes some of instructions while a graphics processing unit (“GPU”)executes other instructions. In at least one embodiment, differentcomponents of a computer system have separate processors and differentprocessors execute different subsets of instructions.

Accordingly, in at least one embodiment, computer systems are configuredto implement one or more services that singly or collectively performoperations of processes described herein and such computer systems areconfigured with applicable hardware and/or software that enableperformance of operations. Further, a computer system that implements atleast one embodiment of present disclosure is a single device and, inanother embodiment, is a distributed computer system including multipledevices that operate differently such that distributed computer systemperforms operations described herein and such that a single device doesnot perform all operations.

Use of any and all examples, or exemplary language (such as “such as”)provided herein, is intended merely to better illuminate embodiments ofdisclosure and does not pose a limitation on scope of disclosure unlessotherwise claimed. No language in specification should be construed asindicating any non-claimed element as essential to practice ofdisclosure.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

In description and claims, terms “coupled” and “connected,” along withtheir derivatives, may be used. It should be understood that these termsmay be not intended as synonyms for each other. Rather, in particularexamples, “connected” or “coupled” may be used to indicate that two ormore elements are in direct or indirect physical or electrical contactwith each other. “Coupled” may also mean that two or more elements arenot in direct contact with each other, but yet still co-operate orinteract with each other.

Unless specifically stated otherwise, it may be appreciated thatthroughout specification, references to processing, computing,calculating, determining, or the like, refer to action and/or processesof a computer or computing system, or similar electronic computingdevice, that manipulate and/or transform data represented as physical,such as electronic, quantities within computing system's registersand/or memories into other data similarly represented as physicalquantities within computing system's memories, registers or other suchinformation storage, transmission or display devices.

In a similar manner, a processor may refer to any device or portion of adevice that processes electronic data from registers and/or memory andtransform that electronic data into other electronic data that may bestored in registers and/or memory. As non-limiting examples, “processor”may be a CPU or a GPU. A “computing platform” may include one or moreprocessors. As used herein, “software” processes may include, forexample, software and/or hardware entities that perform work over time,such as tasks, threads, and intelligent agents. Also, each process mayrefer to multiple processes, for carrying out instructions in sequenceor in parallel, continuously or intermittently. Terms “system” and“method” are used herein interchangeably insofar as system may embodyone or more methods and methods may be considered a system.

In present document, references may be made to obtaining, acquiring,receiving, or inputting analog or digital data into a subsystem,computer system, or computer-implemented machine. Obtaining, acquiring,receiving, or inputting analog and digital data can be accomplished in avariety of ways such as by receiving data as a parameter of a functioncall or a call to an application programming interface. In someimplementations, process of obtaining, acquiring, receiving, orinputting analog or digital data can be accomplished by transferringdata via a serial or parallel interface. In another implementation,process of obtaining, acquiring, receiving, or inputting analog ordigital data can be accomplished by transferring data via a computernetwork from providing entity to acquiring entity. References may alsobe made to providing, outputting, transmitting, sending, or presentinganalog or digital data. In various examples, process of providing,outputting, transmitting, sending, or presenting analog or digital datacan be accomplished by transferring data as an input or output parameterof a function call, a parameter of an application programming interfaceor interprocess communication mechanism.

Although discussion above sets forth example implementations ofdescribed techniques, other architectures may be used to implementdescribed functionality, and are intended to be within scope of thisdisclosure. Furthermore, although specific distributions ofresponsibilities are defined above for purposes of discussion, variousfunctions and responsibilities might be distributed and divided indifferent ways, depending on circumstances.

Furthermore, although subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that subject matter claimed in appended claims is notnecessarily limited to specific features or acts described. Rather,specific features and acts are disclosed as exemplary forms ofimplementing the claims.

What is claimed is:
 1. A cooling system for a datacenter, comprising: anevaporative cooling subsystem to provide blown air for the datacenteraccording to a first heat feature in the datacenter, and to work with arepurposable refrigerant cooling subsystem that controls moisture of theblown air in a first configuration of the repurposable refrigerantcooling subsystem, the repurposable refrigerant cooling subsystem tocool the datacenter in a second configuration and according to a secondheat feature in the datacenter.
 2. The cooling system of claim 1,further comprising: the repurposable refrigerant cooling subsystem tofunction in the second configuration when the evaporative coolingsubsystem is ineffective in achieving a reduction in heat generated inat least one server or rack of the datacenter.
 3. The cooling system ofclaim 1, further comprising: the evaporative cooling subsystem and therepurposable refrigerant cooling subsystem located within aninfrastructure area of a trailer to provide a determined range ofcooling capacities to support mobility of the datacenter.
 4. The coolingsystem of claim 1, further comprising: a refrigerant-based heat transfersubsystem to enable one or more of the evaporative cooling subsystem,the repurposable refrigerant cooling subsystem, and a dielectric-basedcooling subsystem, each having different cooling capabilities for thedatacenter.
 5. The cooling system of claim 4, further comprising: theevaporative cooling subsystem having associated first looping componentsto guide a first cooling media to at least one first rack of thedatacenter according to a first heat feature or a first coolingrequirement of the at least one first rack; and the repurposablerefrigerant cooling subsystem having associated second loopingcomponents to guide a second cooling media to the at least one firstrack of the datacenter according to a second heat feature or a secondcooling requirement of the at least one first rack.
 6. The coolingsystem of claim 5, wherein the first heat feature or a first coolingrequirement is within a first threshold of the different coolingcapacities and second heat feature or a second cooling requirement iswithin a second threshold of the different cooling capacities.
 7. Thecooling system of claim 5, wherein the first cooling media is the blownair and the second cooling is a refrigerant.
 8. The cooling system ofclaim 1, further comprising: one or more humidity sensors associatedwith the datacenter to monitor humidity of the blown air; and one ormore controllers having at least one processor and memory includinginstructions to execute on the at least one process to cause therepurposable refrigerant cooling subsystem to control the humidity ofthe blown air.
 9. The cooling system of claim 1, further comprising: oneor more servers having an air distribution unit and a refrigerantdistribution unit to enable different cooling media to address differentcooling requirements of the one or more servers reflected by the firstheat feature and the second heat feature.
 10. The cooling system ofclaim 1, further comprising: a learning subsystem comprising at leastone processor to evaluate temperatures within servers or one or moreracks in the datacenter with flow rates of different cooling media, andto provide outputs associated with at least two temperatures tofacilitate movement of two cooling media from the cooling system bycontrolling flow controllers associated with the two cooling media toaddress the first heat feature and the second heat feature.
 11. Thecooling system of claim 10, further comprising: cold plates associatedwith components in the servers; the flow controllers facilitating themovement of the two cooling media through the cold plates or through theservers; and the learning subsystem executing a machine learning modelto: process the temperatures using multiple neuron levels of the machinelearning model having the temperatures and having prior associated flowrates for the two cooling media; and provide the outputs associated withflow rates for the two cooling media to the flow controllers, theoutputs provided after an evaluation of the prior associated flow ratesof individual cooling media of the two cooling media and priorassociated temperatures.
 12. The cooling system of claim 1, furthercomprising: one or more first flow controllers maintaining a firstcooling media associated with the evaporative cooling subsystem at afirst flow rate and maintaining a second cooling media associated withthe repurposable refrigerant cooling subsystem at a second flow rate,the first flow rate based in part on the first heat feature; and one ormore second flow controllers maintaining the second cooling mediaassociated with the repurposable refrigerant cooling subsystem at athird flow rate based in part on the second heat feature, the one ormore first flow controllers deactivated based in part on the second heatfeature.
 13. At least one processor for a cooling system, comprising: atleast one logic unit to control flow controllers associated with arepurposable refrigerant cooling subsystem and an evaporative coolingsubsystem based in part on a first heat feature in the datacenter andbased in part on a second heat feature in the datacenter, theevaporative cooling subsystem to work with the repurposable refrigerantcooling subsystem to control moisture of blown air from the evaporativecooling subsystem in a first configuration of the repurposablerefrigerant cooling subsystem, and the repurposable refrigerant coolingsubsystem to cool the datacenter in a second configuration of therepurposable refrigerant cooling subsystem, the blown air and thecooling of the datacenter according to the first heat feature and thesecond heat feature respectively.
 14. The least one processor of claim13, further comprising: a learning subsystem to evaluate temperatureswithin servers or one or more racks in the datacenter with flow rates ofdifferent cooling media, and to provide outputs associated with at leasttwo temperatures to facilitate movement of two cooling media from thecooling system by controlling the flow controllers associated with thetwo cooling media to address the first heat feature and the second heatfeature.
 15. The least one processor of claim 14, further comprising:the learning subsystem executing a machine learning model to: processthe temperatures using multiple neuron levels of the machine learningmodel having the temperatures and having prior associated flow rates forthe two cooling media; and provide the outputs associated with flowrates for the two cooling media to the flow controllers, the outputsprovided after an evaluation of the prior associated flow rates ofindividual cooling media of the two cooling media and prior associatedtemperatures.
 16. The at least one processor of claim 14, furthercomprising: instruction outputs to communicate the outputs associatedwith the flow controllers to enable a first flow rate of a firstindividual cooling media used to address the first heat feature in thedatacenter, while maintaining a second flow rate of a second individualcooling media to control humidity of the first individual cooling media;and to deactivate the first individual cooling media and to enable athird flow rate for the second individual cooling media used to addressthe second heat feature in the datacenter.
 17. The at least oneprocessor of claim 14, further comprising: the at least one logic unitadapted to receive temperature values from temperature sensors andhumidity values from humidity sensors, the temperature sensors and thehumidity sensors associated with the servers or the one or more racks,and adapted to facilitate movement of a first individual cooling mediaand a second individual cooling media concurrently to address the firstheat feature of the data center, and individually to address the secondheat feature of the datacenter.
 18. At least one processor for a coolingsystem, comprising: at least one logic unit to train one or more neuralnetwork having hidden layers of neurons to evaluate temperatures, flowrates, and humidity that are associated with a repurposable refrigerantcooling subsystem and an evaporative cooling subsystem based in part ona first heat feature in the datacenter and based in part on a secondheat feature in the datacenter, the evaporative cooling subsystem towork with the repurposable refrigerant cooling subsystem to controlmoisture of blown air from the evaporative cooling subsystem in a firstconfiguration of the repurposable refrigerant cooling subsystem, and therepurposable refrigerant cooling subsystem to cool the datacenter in asecond configuration of the repurposable refrigerant cooling subsystem,the blown air and the cooling of the datacenter according to the firstheat feature and the second heat feature respectively.
 19. The at leastone processor of claim 18, further comprising: the at least one logicunit to output at least one instruction associated with at least twotemperatures to facilitate movement of the at least two cooling media bycontrolling flow controllers associated with the at least two coolingmedia to facilitate flowing of the two cooling media concurrently in thefirst configuration of the repurposable refrigerant cooling subsystemand to facilitate flowing of one of the two cooling media alone in thesecond configuration of the repurposable refrigerant cooling subsystem.20. The at least one processor of claim 19, further comprising: at leastone instruction output to communicate the output associated with flowcontrollers to enable a first flow rate of a first individual coolingmedia used to address the first heat feature in the datacenter, whilemaintaining a second flow rate of a second individual cooling media tocontrol humidity of the first individual cooling media; and todeactivate the first individual cooling media and to enable a third flowrate for the second individual cooling media used to address the secondheat feature in the datacenter.
 21. The at least one processor of claim18, further comprising: the at least one logic unit adapted to receivetemperature values from temperature sensors and humidity values fromhumidity sensors, the temperature sensors and the humidity sensorsassociated with servers or one or more racks in the datacenter, andadapted to facilitate movement of a first individual cooling media and asecond individual cooling media concurrently to address the first heatfeature of the data center, and individually to address the second heatfeature of the datacenter.
 22. A datacenter cooling system, comprising:at least processor to train one or more neural network having hiddenlayers of neurons for evaluating temperatures and flow rates that areassociated with at least two cooling media of a repurposable refrigerantcooling subsystem comprising at least an evaporative cooling subsystemand the repurposable refrigerant cooling subsystem that are rated fordifferent cooling capacities and adjustable to different coolingrequirements of the datacenter within a minimum and a maximum of thedifferent cooling capacities, the cooling system further having a formfactor for one or more racks of a datacenter.
 23. The datacenter coolingsystem of claim 22, further comprising: the at least one processor tooutput at least one instruction associated with at least twotemperatures to facilitate movement of the at least two cooling media bycontrolling flow controllers associated with the at least two coolingmedia to facilitate flowing of the two cooling media concurrently in thefirst configuration of the repurposable refrigerant cooling subsystemand to facilitate flowing of one of the two cooling media alone in thesecond configuration of the repurposable refrigerant cooling subsystem.24. The datacenter cooling system of claim 23, further comprising: atleast an instruction output of the at least one processor to communicatethe output associated with flow controllers to enable a first flow rateof a first individual cooling media used to address the first heatfeature in the datacenter, while maintaining a second flow rate of asecond individual cooling media to control humidity of the firstindividual cooling media; and to deactivate the first individual coolingmedia and to enable a third flow rate for the second individual coolingmedia used to address the second heat feature in the datacenter.
 25. Thedatacenter cooling system of claim 22, further comprising: the at leastone processor adapted to receive temperature values from temperaturesensors and humidity values from humidity sensors, the temperaturesensors and the humidity sensors associated with servers or one or moreracks in the datacenter, and adapted to facilitate movement of a firstindividual cooling media and a second individual cooling mediaconcurrently to address the first heat feature of the data center, andindividually to address the second heat feature of the datacenter.
 26. Amethod for cooling a datacenter, comprising: providing an evaporativecooling subsystem to provide blown air for the datacenter according to afirst heat feature in the datacenter, and to work with a repurposablerefrigerant cooling subsystem that controls moisture of the blown air ina first configuration of the repurposable refrigerant cooling subsystem,the repurposable refrigerant cooling subsystem to cool the datacenter ina second configuration according to a second heat feature in thedatacenter.
 27. The method of claim 26, further comprising: guiding theblown air to at least one first rack of the datacenter, from theevaporative cooling subsystem, using first looping components andaccording to a first heat feature or a first cooling requirement of theat least one first rack; and guiding a refrigerant to the at least onefirst rack of the datacenter, from the repurposable refrigerant coolingsubsystem, using second looping components and according to a secondheat feature or a second cooling requirement of the at least one firstrack.
 28. The method of claim 26, further comprising: enabling therepurposable refrigerant cooling subsystem to function in the secondconfiguration when the evaporative cooling subsystem is ineffective inachieving a reduction in heat generated in at least one server or rackof the datacenter.
 29. The method of claim 26, further comprising:enabling one or more of the evaporative cooling subsystem, therepurposable refrigerant cooling subsystem, and a dielectric-basedcooling subsystem, each having different cooling capabilities for thedatacenter, using a refrigerant-based heat transfer subsystem.
 30. Themethod of claim 26, further comprising: monitoring humidity of the blownair using one or more humidity sensors associated with the datacenter;and causing the repurposable refrigerant cooling subsystem to controlthe humidity of the blown air.